Information system, data transfer method and data protection method

ABSTRACT

Availability of an information system including a storage system that performs remote copy between two or more storage apparatuses and a host computer using such storage system is improved. A third storage apparatus including a third volume is coupled to a first storage apparatus, a fourth storage apparatus including a fourth volume is coupled to a second storage apparatus, the first and third storage apparatuses perform remote copy of copying data stored in a first volume to the third volume, the first and second storage apparatuses perform remote copy of copying data stored in the first volume to a second volume, and the third and fourth storage apparatuses perform remote copy of copying data stored in the third volume to the fourth volume.

CROSS REFERENCES

This application relates to and claims priority from Japanese PatentApplications No. 2007-85675, filed on Mar., 28, 2007 and No.2006-293485, filed on Oct. 30, 2006, the entire disclosure of which isincorporated herein by reference.

BACKGROUND

The present invention relates to a storage system comprising a pluralityof storage areas, and a host computer coupled to the storage system.

Generally, an information system is equipped with a storage apparatusthat uses an HDD (hard disk drive) as a storage device, and a storagesystem including this storage apparatus is accessed from a plurality ofhost systems (hosts, for example) via a storage area network (SAN:Storage Area Network). Generally, with a storage apparatus, ahigh-reliability method according to RAID (Redundant Array ofIndependent (or Inexpensive) Disks) technology is adopted to providereliability to the storage apparatus beyond the reliability of astand-alone HDD. Nevertheless, pursuant to the advancement ofinformation society in recent years, the availability (servicecontinuity) of information systems depending on reliability based onRAID is becoming inadequate.

Japanese Patent Laid-Open Publication No. H7-244597 (Patent Document 1)describes high-availability technology to deal with the foregoingsituation. This technology prepares a production site and a backup siterespectively including a host computer (hereinafter abbreviated as a“host”) and a storage apparatus, and mirrors data stored in the storageapparatus of the production site to the storage apparatus of the backupsite. If the storage apparatus of the production site fails and shutsdown, application processing that was suspended as a result of suchstorage apparatus failure is resumed using the storage apparatus and thehost of the backup site. This technology is generally referred to asremote copy or remote mirroring.

SUMMARY

With the technology of Patent Document 1, since the application isresumed with a different host when a storage apparatus fails and shutsdown, re-boot processing of the application is required. Needless tosay, there will be a problem concerning availability since theapplication will not be able to perform its normal operation from thetime such application is suspended until the re-boot is complete.

Thus, an object of the present invention is to improve the availabilityof an information system including a storage system that performs remotecopy between two or more storage apparatuses, and a host that uses thisstorage system.

In order to achieve the foregoing object, the present invention providesan information system having a first host computer as a host system, afirst storage apparatus coupled to the first host computer and includinga first volume, and a second storage apparatus coupled to the firststorage apparatus and the first host computer and including a secondvolume. This information system comprises a third storage apparatuscoupled to the first storage apparatus and including a third volume, anda fourth storage apparatus coupled to the second storage apparatus andincluding a fourth volume. The first and second storage apparatusesexecute remote copy of copying data stored in the first volume to thesecond volume. The first and third storage apparatuses execute remotecopy of copying data stored in the first volume to the third volume. Thesecond and fourth storage apparatuses execute remote copy of coping datastored in the second volume to the fourth volume.

The present invention also provides a data protection method in aninformation system having a first host computer as a host system, afirst storage apparatus coupled to the first host computer and includinga first volume, and a second storage apparatus coupled to the firststorage apparatus and the first host computer and including a secondvolume. This data protection method comprises a first step of connectinga third storage apparatus including a third volume to the first storageapparatus and connecting a fourth storage apparatus including a fourthvolume to the second storage apparatus, and the first and second storageapparatuses executing remote copy of copying data stored in the firstvolume to the second volume, and a second step of the first and thirdstorage apparatuses executing remote copy of copying data stored in thefirst volume to the third volume, and the second and fourth storageapparatuses executing remote copy of copying data stored in the secondvolume to the fourth volume.

The present invention further provides an information system having afirst host computer as a host system, a first storage apparatus coupledto the first host computer and including a first volume, and a secondstorage apparatus coupled to the first storage apparatus and the firsthost computer and including a second volume. This information systemcomprises a third storage apparatus coupled to the first and secondstorage apparatuses and including a third volume, and a fourth storageapparatus coupled to the third storage apparatus and including a fourthvolume. The first and second storage apparatuses execute remote copy ofcopying data stored in the first volume to the second volume. The firstand third storage apparatuses execute remote copy of copying data storedin the first volume to the third volume. The third and fourth storageapparatuses execute remote copy of coping data stored in the thirdvolume to the fourth volume.

The present invention additionally provides a data protection method inan information system having a first host computer as a host system, afirst storage apparatus coupled to the first host computer and includinga first volume, and a second storage apparatus coupled to the firststorage apparatus and the first host computer and including a secondvolume. This data protection method comprises a first step of connectinga third storage apparatus including a third volume to the first andsecond storage apparatuses and connecting a fourth storage apparatusincluding a fourth volume to the third storage apparatus, the first andsecond storage apparatuses executing remote copy of copying data storedin the first volume to the second volume, and the first and thirdstorage apparatuses executing remote copy of copying data stored in thefirst volume to the third volume, and a second step of the third andfourth storage apparatuses executing remote copy of copying data storedin the third volume to the fourth volume.

According to the present invention, it is possible to improve theavailability of an information system including a storage system thatperforms remote copy between two or more storage apparatuses, and a hostthat uses this storage system.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram showing an example of the hardwareconstitution of an information system according to a first embodiment ofthe present invention;

FIG. 2 is a first conceptual diagram showing the overview of a firstembodiment of the present invention;

FIG. 3 is a second conceptual diagram showing the overview of the firstembodiment;

FIG. 4 is a third conceptual diagram showing the overview of the firstembodiment;

FIG. 5 is a conceptual diagram representing the software constitution ina host;

FIG. 6 is a block diagram representing the software constitution in avirtual storage apparatus and a storage apparatus;

FIG. 7 is a conceptual diagram representing the pair status of remotecopy and the transition of pair status;

FIG. 8 is a conceptual diagram showing a device relation table to bemanaged by an I/O path manager;

FIG. 9 is a flowchart showing the flow when the I/O path managerperforms initialization processing;

FIG. 10 is a flowchart showing the flow when the I/O path managerperforms write processing;

FIG. 11 is a flowchart showing the flow when the I/O path managerperforms read processing;

FIG. 12 is a conceptual diagram showing the overview of a secondembodiment of the present invention;

FIG. 13 is a conceptual diagram showing the overview of a thirdembodiment of the present invention;

FIG. 14 is a conceptual diagram showing the overview of a fourthembodiment of the present invention;

FIG. 15 is a conceptual diagram showing the overview of a fifthembodiment of the present invention;

FIG. 16 is a conceptual diagram showing the overview of a sixthembodiment of the present invention;

FIG. 17 is a conceptual diagram showing the overview of a seventhembodiment of the present invention;

FIG. 18 is a conceptual diagram showing the overview of a eighthembodiment of the present invention;

FIG. 19 is a conceptual diagram showing the overview of a ninthembodiment of the present invention;

FIG. 20 is a conceptual diagram showing the overview of a tenthembodiment of the present invention;

FIG. 21 is a conceptual diagram showing the overview of a eleventhembodiment of the present invention;

FIG. 22 is a conceptual diagram showing the overview of a twelfthembodiment of the present invention;

FIG. 23 is a conceptual diagram showing the overview of a thirteenthembodiment of the present invention;

FIG. 24 is a conceptual diagram showing the overview of a fourteenthembodiment of the present invention;

FIG. 25 is a flowchart showing a different flow when the I/O pathmanager performs write processing;

FIG. 26 is a flowchart showing a different flow when the I/O pathmanager performs read processing;

FIG. 27 is a flowchart showing a pair operation according to a writerequest to be performed by the storage apparatus when the I/O pathmanager is to perform the write processing of FIG. 25;

FIG. 28 is a conceptual diagram showing the overview of a fifteenthembodiment of the present invention;

FIG. 29 is a conceptual diagram showing the overview of a sixteenthembodiment of the present invention;

FIG. 30 is a conceptual diagram showing the overview of the sixteenthembodiment;

FIG. 31 is a block diagram representing the software constitution of thevirtual storage apparatus and the storage apparatus according to anembodiment of the present invention;

FIG. 32 is a flowchart showing a different flow when the virtual storageapparatus performs write processing;

FIG. 33 is a flowchart showing a different flow when the virtual storageapparatus performs read processing;

FIG. 34 is a flowchart showing the flow of staging processing for AOU;

FIG. 35 is a conceptual diagram explaining the specific contents of theAOU address mapping information;

FIG. 36 is a conceptual diagram explaining the specific contents of theAOU pool management information;

FIG. 37 is a block diagram showing an example of the hardwareconstitution of the information system according to a seventeenthembodiment;

FIG. 38 is a first conceptual diagram showing an overview of theseventeenth embodiment;

FIG. 39 is a second conceptual diagram showing an overview of theseventeenth embodiment;

FIG. 40 is a third conceptual diagram showing an overview of theseventeenth embodiment;

FIG. 41 is a block diagram representing the software constitution in alocal-side storage apparatus and a remote-side storage apparatus;

FIG. 42 is a conceptual diagram explaining the structure of a journalvolume;

FIG. 43 is a conceptual diagram representing the pair status of remotecopy and the transition of pair status according to the seventeenthembodiment;

FIG. 44 is a flowchart showing the flow of initialization processingaccording to the seventeenth embodiment;

FIG. 45 is a flowchart showing the flow of failover processing accordingto the seventeenth embodiment;

FIG. 46 is a conceptual diagram explaining update information;

FIG. 47 is a conceptual diagram explaining journal group information41330P;

FIG. 48 is a flowchart showing the flow of JNLRD processing;

FIG. 49 is a flowchart showing the flow of journal creation processing;

FIG. 50 is a flowchart showing the flow of JNLRD processing;

FIG. 51 is a flowchart showing the flow of restoration processing;

FIG. 52 is a first conceptual diagram showing an overview of aneighteenth embodiment;

FIG. 53 is a second conceptual diagram showing an overview of theeighteenth embodiment;

FIG. 54 is a third conceptual diagram showing an overview of theeighteenth embodiment;

FIG. 55 is a flowchart showing the flow of switch processing in remotecopy write processing;

FIG. 56 is a flowchart showing the flow of switch processing in remotecopy read processing;

FIG. 57 is a flowchart showing the flow of switch processing to beperformed in a local-side old secondary storage apparatus;

FIG. 58 is a flowchart showing the flow of switch processing to beperformed in a remote-side primary storage apparatus;

FIG. 59 is a flowchart showing the flow of switch processing to beperformed in a local-side old primary storage apparatus;

FIG. 60 is a flowchart showing the flow of secondary journal releaseprocessing; and

FIG. 61 is a flowchart showing the flow of failover processing accordingto the eighteenth embodiment.

DETAILED DESCRIPTION

Embodiments of the present invention are now explained with reference tothe attached drawings.

(1) First Embodiment

<1. Constitution of Information System>

FIG. 1 is a diagram showing an example of the hardware constitution(configuration) of an information system according to an embodiment ofthe present invention.

The information system, for example, comprises a storage apparatus 1500,a host computer (hereafter abbreviated as a “host”) 1100, a managementhost 1200, and two or more virtual storage apparatuses 1000. A pluralityof storage apparatuses 1500, host computers (hereafter abbreviated asthe “hosts”) 1100, and management hosts 1200 may be provided,respectively. The virtual storage apparatus 1000 and the host 1100 aremutually connected via an I/O network 1300. The virtual storageapparatus 1000 and the storage apparatus 1500 and the management host1200 are mutually connected via a management network (not shown) or theI/O network 1300.

The host 1100 has a host internal network 1104, and coupled to thisnetwork 1104 are a processor (abbreviated as Proc in the diagrams) 1101,a memory (abbreviated as Mem in the diagrams) 1102, and an I/O port(abbreviated as I/O P in the diagrams) 1103. The management host 1200may also have the same hardware constitution as the host 1100.Incidentally, an expansion card for adding an I/O port to the host 1100is sometimes referred to as an HBA (Host Bus Adapter).

The management host 1200 has a display device, and this display deviceis able to display a screen for managing the virtual storage apparatus1000 and the storage apparatus 1500. Further, the management host 1200is able to receive a management operation request from a user (forinstance, an operator of the management host 1200), and send thereceived management operation request to the virtual storage apparatus1000 and the storage apparatus 1500. The management operation request isa request for operating the virtual storage apparatus 1000 and thestorage apparatus 1500, and, for example, there are a parity groupcreation request, an internal LU (Logical Unit) creation request, a pathdefinition request, and operations related to a virtualization function.

Connection via a fibre channel is foremost considered as the I/O network1300, but in addition thereto, a combination of FICON (Flbre CONnection:registered trademark), or Ethernet (registered trademark) and TCP/IP(Transmission Control Protocol/Internet Protocol) and iSCSI (internetSCSI (Small Computer System Interface)), and a combination of networkfile systems such as Ethernet (registered trademark) and NFS (NetworkFile System) of CIFS (Common Internet File System) may also beconsidered. Further, the I/O network 1300 may also be other than theabove so as long as it is a communication device capable of transferringI/O requests. Further, the network that connects the virtual storageapparatus 1000 and the storage apparatus 1500 is also the same as theI/O network 1300.

The virtual storage apparatus 1 000 comprises a controller (indicated asCTL in the diagrams) 1010, a cache memory (indicated as CM in thediagrams) 1020, and a plurality of HDDs 1030. As a preferred embodiment,the controller 1010 and the cache memory 1020 are respectivelyconstituted of a plurality of components. The reason for this is becauseeven if a failure occurs in a single component and such component isblocked, the remaining components can be used to continue receiving I/Orequests as represented by read and write requests.

The controller 1010 is an apparatus (a circuit board, for example) forcontrolling the operation of the virtual storage apparatus 1000. Thecontroller 1010 has an internal network 1017, and coupled to thisinternal network 1017 are an I/O port 1013, a cache port (abbreviated asCP in the diagrams) 1015, a management port (abbreviated as MP in thediagrams) 1016, a back-end port (abbreviated as B/E P in the diagrams)1014, a processor (a CPU (Central Processing Unit), for instance) 1011,and a memory 1012. The controllers 1010 and the cache memories 1020 aremutually connected each other via a storage internal network 1050.Further, the controller 1010 and the respective HDDs 1030 are mutuallyconnected via a plurality of back-end networks 1040.

The hardware constitution of the storage apparatus 1500 is constitutedof similar components as those of the virtual storage apparatus 1000.Incidentally, when the virtual storage apparatus 1000 is a dedicateddevice or switch for virtualization without an HDD, the storageapparatus 1500 does not need to be constituted of similar components asthose of the virtual storage apparatus 1000. Further, the internalnetwork of the host 1100 and the virtual storage apparatus 1000 ispreferably of a broader bandwidth than the transfer bandwidth of the I/Oport 1013, and all or a part thereof may be substituted with a bus orswitch-type network. Further, in FIG. 1, although only one I/O port 1013is provided to the controller 1010, in reality, a plurality of I/O ports1013 may exist in the controller 1010.

According to the foregoing hardware constitution, the host 1100 will beable to read or write all or a part of the data stored in the HDD of thevirtual storage apparatus 1000 and the storage apparatus 1500.Incidentally, in the ensuing explanation, the system handling thestorage of data is referred to as a storage cluster. Further, asubsystem that realizes high availability by including two subsystemsinside the storage cluster and which includes the virtual storageapparatus 1000 and/or the storage apparatus 1500 is referred to as astorage subsystem.

<2. Overview of Present Embodiment>

In this embodiment, in order to improve the availability of a storagesystem including the virtual storage apparatus 1000 having avirtualization function for virtualizing a storage area such as a volumein another storage apparatus, a redundant constitution using anothervirtual storage apparatus 1000 is adopted. FIG. 2 is a diagram showingan overview of such a duplex constitution.

In this overview, the storage system includes a virtual storageapparatus 1000L, a virtual storage apparatus 1000R, a storage apparatus1500L, and a storage apparatus 1500R. Incidentally, in order to simplifythe following explanation, let it be assumed that the virtual storageapparatus 1000L and the storage apparatus 1500L serve as a primarysystem (production system), and the virtual storage apparatus 1000R andthe storage apparatus 1500R serve as a secondary system (backup system).Nevertheless, when the number of volumes to be respectively provided bythe virtual storage apparatuses 1000L, 1000R to the host 1100 is two ormore volumes, in substitute for handling the primary system/secondarysystem in virtual storage apparatus units, only the virtual storageapparatuses 1000L, 1000R to serve as the primary system in volume unitsneed to be defined.

The respective virtual storage apparatuses 1000L, 1000R provide partialor all areas of a parity group (configured based on RAID technology)with its own HDD 1030 as the constituent element as a volume 3000LA anda volume 3000RA to the host 1100 (corresponds to the portion in which‘A’ is indicated in a cylinder in FIG. 2). Further, the virtual storageapparatus 1000 is also able to optionally provide, based on thevirtualization function, virtual volumes 3000LB, 3000RB (volumes inwhich the nonvolatile storage areas of the corresponding HDD or the likeexist outside the virtual storage apparatuses 1000L, 1000R). In thisoverview, a part or all of the volumes 3500LB, 3500RB provided by thestorage apparatuses 1500L, 1500R are used as the correspondingnonvolatile storage areas. Incidentally, reference to “data of a volume”in the following explanation includes, in addition to the data stored inthe HDD 1030, data that is temporarily stored in the cache memory 1020.Further, “data of a virtual volume” described later includes, inaddition to the data stored in the volumes 3500LB, 3500RB of the storageapparatuses 1500L, 1500R, data that is temporarily stored in the cachememory 1020 of the virtual storage apparatuses 1000L, 1000R.

Meanwhile, an application program (hereinafter sometimes abbreviated asan “application”) 2010, an OS, and system programs as represented bydaemon and management programs for assisting in the setting andprocessing of the OS are executed in the host 1100. The OS provides tothe application 2010 an interface for I/O requests to data existing inthe volumes 3000LA, 3000LB, 3000RA, 3000RB provided by the virtualstorage apparatuses 1000L, 1000R, and sends I/O requests to theappropriate virtual storage apparatuses 1000L, 1000R and volumes 3000LA,3000LB, 3000RA, 3000RB according to the request from the application2010. In a normal status, the host 1100 issues an I/O request asrepresented by a read or write request to the volumes 3000LA, 3000LB ofthe virtual storage apparatus 1000L, and thereby sends and receivesdata. In other words, upon receiving a read request, the virtual storageapparatus 1000L reads data from the HDD 1030 and returns such data tothe host 110 when the requested volumes 3000LA, 3000LB, 3500LBcorrespond to the HDD 1030 inside the virtual storage apparatus 1000L,or acquires the necessary data and returns such data (all or a part) tothe host 1100 by issuing a read request to the storage apparatus 1500L.

In the case of a write request, in order to make the data redundant, thevirtual storage apparatus 1000L that received the write data sends thewrite data to the virtual storage apparatus 1000R as the secondarysystem, and returns the write complete message to the host 1100 afterthe virtual storage apparatus 1000L receives a write data receptioncomplete message from the virtual storage apparatus 1000R. Incidentally,write data to the virtual storage apparatus 1000L and write datareceived by the virtual storage apparatus 1000R via the virtual storageapparatus 1000L may also be temporarily retained in the cache memories1020L, 1020R of the respective virtual storage apparatuses 1000L, 1000R.Incidentally, as one example of this embodiment, the transfer of thiswrite data is conducted via storage remote copy.

FIG. 3 shows the processing overview of the information system after afailure occurred in the virtual storage apparatus 1000L under a normalstatus.

When the primary virtual storage apparatus 1000L fails and shuts down,the system program in the host 1100 detects this failure, and switchesthe destination of the I/O request from the primary virtual storageapparatus 1000L to the secondary virtual storage apparatus 1000R.Nevertheless, in this case also, the application 2010 is able tocontinue I/O without being aware that the destination of the I/O requesthas been switched. Thus, normally, as a volume identifier designated atthe time of an I/O request from the application 2010 or the file system,the system program provides a virtual volume identifier (or a devicefile) at an OS layer (more specifically, a layer that is lower than thefile system), and the lower layer of OS manages the correspondence ofthat identifier and the identifier (or device file) actually allocatedto the volume. When switching the destination of the I/O request, thecorrespondence thereof is switched from the volume 3000LA and the volume3000LB of the virtual storage apparatus 1000L to the volume 3000RA andthe volume 3000RB of the virtual storage apparatus 1000R, so as torealize switching that will be transparent to the application 2010.

Further, the virtual storage apparatus 1000R is also able to process thewrite request, according to the arrival of such write request to thevolumes 3000RA, 3000RB from the host 1100, or other express fail overrequests. As an example of this change processing, in line with the datacopy from the virtual storage apparatus 1000L to the virtual storageapparatus 1000R, when the setting is configured to deny the writerequest from the host 1100 to the volumes 3000RA, 3000RB of the virtualstorage apparatus 1000R, such setting is cancelled. Further, when writedata is being transferred using remote copy, the copy status of remotecopy may also be changed.

FIG. 4 shows the processing overview of the information system after theoccurrence of a failure in the network between the virtual storageapparatuses 1000L, 1000R.

The virtual storage apparatus 1000L that detected the network failurenotifies this failure to the host 1100. The host 1100 that received thefailure notice requests the secondary virtual storage apparatus 1000R toprocess the write request and issues subsequent write requests to boththe primary virtual storage apparatus 1000L and the secondary virtualstorage apparatus 1000R so as to make the data of the primary system andthe data of the secondary system uniform.

<3. Programs and Information to be Executed by Host 1100>

FIG. 5 is a diagram illustrating the concept to be provided by therespective software programs in addition to the software programs to beexecuted in the host 1100 and information to be used by such softwareprograms. Incidentally, although the software programs are retained inthe memory 1102 (FIG. 1) and executed by the processor 1101 (FIG. 1),such software programs may be partially realized as hardware andexecuted.

In the host 1100, in addition to the application 2010 and the remotecopy manager 5030, a file system 5020, an I/O path manager 5000 and anHBA device driver 5010 are executed as program modules inside the OS orKernel (it is not necessary to execute all processing, for the filesystem 5020, the I/O path manager 5000 or the HBA device driver 5010,inside the Kernel.).

The HBA device driver 5010 is a program for sending and receiving I/Orequests and incidental data through the I/O port 1103 (FIG. 1) mountedon the HBA, and controlling communication with the other virtual storageapparatuses 1000L, 1000R and the storage apparatuses 1500L, 1500R. TheHBA device driver 5010 is also able to provide an identifiercorresponding to the volumes 3000LA, 3000LB, 3000RA, 3000RB provided bythe virtual storage apparatuses 1000L, 1000R to the upper layer, andreceive an I/O request accompanied with such identifier. The volume 5040illustrates this concept, and corresponds to the respective volumes3000LA, 3000LB, 3000RA, 3000RB provided by the virtual storageapparatuses 1000L, 1000R.

The I/O path manager 5000 is a module for switching the I/O requestdestination of the application 2010. This module provides to the filesystem 5020 an I/O request interface and the identifier, which is thesame type of identifier corresponding to the volume 5040 provided by theHBA device driver 5010 and corresponds to a virtual volume in the host1100. The identifier corresponding to the virtual volume in the host1100 corresponds to the identifier corresponding to the volume 5040provided by the HBA device driver 5010 in the module, and the devicerelation table 5001 retains the correspondence thereof. The volume 5050illustrates the concept of this virtual volume in the host 1100, and, inFIG. 5, an example of the correspondence thereof corresponds to theidentifier corresponding to the volumes 3000LA, 3000LB of the virtualstorage apparatus 1000L (to put it differently, it could be said thatthe entities of the virtual volume 5050 in the host 1100 are the volumes3000LA, 3000LB of the virtual storage apparatus 1000L).

An I/O request up to this layer is usually designated in a fixed-lengthblock access format. Nevertheless, the I/O request is not limitedthereto when the host 1100 is a mainframe, and it may also be designatedin a CKD (Count Key Data) format.

The file system 5020 is a module for sending an I/O request and sendingand receiving data from/to the virtual storage apparatuses 1000L, 1000R,which is done through the identifier and the I/O interface correspondingto the volume 5040 provided by the HBA device driver 5010, and theidentifier and the interface corresponding to the virtual volume 5050 inthe host 1100 provided by the I/O path manager 5000. FIG. 5 illustratesas an example of the structure of a directory tree inside the filesystem 5020 in a state where a part of such tree structure 5052 isstored in the volume 5050 provided through virtualization in the host1100 by the I/O path manager 5000 (as explained above, more precisely,provision of the virtual volume 5050 in the host 1100 of the I/O pathmanager 5000 is made through the identifier, and the data indicated asbeing stored in the volume 5050 is actually stored in the volumes3000LA, 3000LB, 3000RA, 3000PB provided by the virtual storageapparatuses 1000L, 1000R shown in the device relation table 5001). Thefile system 5020 provides an interface of a file l/O to the application2010. The file system 5020 called from the application 2010 through thefile I/O interface converts the read or write request accompanied with afile name and data offset in the file into a read or write request of ablock format while referring to structural information in the filesystem 5020 such as a directory file or an inode, and delivers the reador write request to the I/O path manager 5000 or the HBA device driver5010.

Incidentally, with a Unix system or Windows (registered trademark)system OS, the file I/O interface is used to provide a function referredto as a device file system as the interface for directly operating thedata of volumes. Normally, the device file system is deployed under thecontrol of the ‘/dev’ directory of the file space, and the file name ofthe file of the foregoing directory and below (rsda and so on in theillustrated example) corresponds to the volumes 5040, 5050 provided bythe lower layer (HBA device driver 5010 and I/O path manager 5000) ofthe file system 5020. Then, data stored in the volumes 5040, 5050 can beread and written with the file I/O interface as though such data isstored in the device files 5070, 5080. Incidentally, in the exampleshown in FIG. 5, the device file 5070 (rsda, rsdb, rsdc, rsdd)corresponds to the volume 5040 recognized and provided by the HBA devicedriver 5010, and the device file 5080 (vsda, vsdb) corresponds to thevolume 5050 provided by the I/O path manager 5000. These device files5070, 5080 may be used for the purpose of realizing independent dataorganization or buffer management when the application 2010 is adatabase.

The remote copy manager 5030 is a program for acquiring the status ofremote copy for realizing the data transfer between the virtual storageapparatuses 1000L, 1000R, and for the host 1100 and the I/O path manager5000 to perform the operation of remote copy, and communicates with thevirtual storage apparatuses 1000L, 1000R according to the request of aprogram, a user or the I/O path manager 5000 using this program.

Incidentally, as explained above, it would be desirable if the functionsof the HBA device driver 5010 and the I/O path manager 5000 could bepartially or wholly installed and uninstalled as modules inside theKernel. This is because, since the HBA device driver 5020 is a programfor controlling the HBA, it is often provided by the manufacturer of theHBA. Similarly, since the processing of the I/O path manager 5000 isdecided subject to the processing of the virtual storage apparatuses1000L, 1000R, it is possible that some or all of the modules will beprovided by the manufacturer of the virtual storage apparatuses 1000L,1000R. Therefore, as a result of being able to install/uninstall thisprogram, it will be possible to constitute an information system basedon a broad range of combinations of HBA and virtual storage apparatuses1000L, 1000R. Further, with the present invention, since the primarysystem and the secondary system are switched in a manner that istransparent to the application 2010, transparent switching that does notrequire the recompilation or the like of the application 2010 can berealized by executing processing inside the Kernel. Moreover, since theI/O path manager 5000 exists in the intermediate layer of the filesystem 5020 and the HBA device driver 5010, recompilation of the filesystem 5020 is no longer required, and transparency of the file systemis also secured. In addition, the I/O path manager 5000 is able to usethe functions of the HBA device driver 5010.

Further, the following two methods can be considered when the I/O pathmanager 5000 inside the Kernel calls the remote copy manager 5030 orperforming the opposite communication method thereof.

(A) The I/O path manager 5000 creates a virtual volume forcommunication, and the file system 5020 creates this communicationvolume as a device file in the file space. The remote copy manager 5030stands by in a state of periodically executing a read system call to thedevice file. The I/O path manager 5000 receives an I/O request from theremote copy manager 5030, but pends it internally. Then, when it becomesnecessary for this module to send a message to the remote copy manager5030, the I/O path manager 5000 returns the data containing the messagedefined as a return value of the I/O request to the remote copy manager5030 through the file system 5020. Incidentally, the read system callissued by the remote copy manager thereupon will be forced to waitinside the Kernel for a long period of time. If this is not preferable,the I/O path manager 5000 should return data indicating that there is nomessage to the remote copy manager 5030 through the file system 5020after the lapse of a prescribed period of time, and the remote copymanager 5030 that received this message should execute the read systemcall once again.

(B) Unix (registered trademark) domain socket is used and this istreated as a virtual network communication. Specifically, the remotecopy manager 5030 operates one end of the socket, and the I/O pathmanager 5000 operates the remaining end.

Incidentally, in the following explanation, when the I/O path manager5000 is to operate remote copy or refer to the status, let it be assumedthat such operation is conducted by calling the remote copy manager 5030through the foregoing communication.

<4. Programs and Information to be Executed by Virtual Storage Apparatus1000>

FIG. 6 is a diagram showing the programs to be executed by the virtualstorage apparatuses 1000 (1000L, 1000R) and the storage apparatuses 1500(1500L, 1500R), and information to be managed by these programs.Incidentally, although the programs are retained in the memory 1102(FIG. 1) and the cache memory 1020 and executed by the processor 1101(FIG. 1), such programs may be partially constituted as hardware andexecuted.

<4.1. I/O Processing Program 6020, Parity Group Information 6060 andVolume Information 6050>

The parity group information 6060 contains information relating to thefollowing constitution of each parity group.

(1) Identifier of HDD 1030 configuring the parity group. Since aplurality of HDDs 1030 are participating in the parity group, thisinformation exists in a plurality for each parity group.

(2) RAID level

Further, the volume information 6050 contains information relating tothe following configuration of each volume.

(1) Volume capacity

(2) Identifier of the parity group and areas (start address and/or endaddress) in the parity group storing data corresponding to the volume.

The I/O processing program 6020 executes the following processingrelating to the I/O request received from the host 1100 by referring tothe volume information 6050 and the parity group information 6060.

(A) Staging: Copying data stored in the HDD 1030 to the cache memory1020.

(B) Destaging: Copying data stored in the cache memory 1020 to the HDD1030. Incidentally, as the pre-processing thereof, redundant data basedon RAID technology may also be created.

(C) Read processing: Determining whether data corresponding to therequest exists in the cache memory 1020 in response to the read requestreceived from the host 1100. In case of the data corresponding to therequest not existing in the cache memory 1020, staging processing isexecuted to copy the data to the cache memory 1020, and such data issent to the host 1100. Incidentally, in case of such data existing inthe cache memory 1020, this data is sent to the host 1100.

(D) Write processing: Storing the write data received from the host 1100in the cache memory 1020. Incidentally, in case of the free area in thecache memory 1020 not being enough during the processing, destagingprocessing is executed to copy appropriate data to the HDD 1030, and thearea in the cache memory 1020 is thereafter reused. Further, in case ofthe address, of which data is previously stored in the cache memory1020, is included in the target area of the write request, the data ofthe area may sometimes be directly overwritten in the cache memory 1020.

(E) Cache algorithm: Deciding the data in the HDD 1030 to be staged andthe data in the cache memory 1020 to be destaged according to an LRUalgorithm or the like based on the reference frequency or referenceperiod of data in the cache memory 1020.

<4.2. Virtualization Program 6030 and Virtualization Information 6070>

The virtualization information 6070 contains information relating to thefollowing configuration of each virtualization volume.

(1) Following information concerning areas in the volume of the storageapparatus 1500, and address space in the virtual volume as which theforegoing areas is provided to the host 1100. In case of the virtualvolume constituting a plurality of volumes, the following informationwill also exist in a plurality.

(1-1) Identifier of the storage apparatus 1500 (or identifier of theport), identifier of the volume, and areas (start address and endaddress) in the volume, constituted of the virtual volume

(1-2) Areas (start address and end address) in the virtual volume

(2) Capacity of the virtual volume

The virtualization program 6030 is a program for the virtual storageapparatus 1000 to provide a volume to the host 1100 by using the volumeprovided by the storage apparatus 1500. Incidentally, there are thefollowing patterns as the correspondence of the virtual volume providedby the virtualization program 6030 and the relating volume in thestorage apparatus 1500.

(A) A case of using the overall volume in the storage apparatus 1500 asthe storage area of the virtual volume. In this case, capacity of thevirtual volume will be roughly the same capacity as the selected volume(‘roughly same’ is a case of storing the control information andredundant information in a volume of the storage apparatus 1500. Whenthere is no such information, this will be the same capacity).

(B) A case of using a part of the volume in the storage apparatus 1500as the storage area corresponding to the virtualization volume. Here,capacity of the virtual volume will be roughly the same as the areacapacity to be used.

(C) A case of combining and using a plurality of volumes in a pluralityof storage apparatuses 1500 as the storage area of the virtual volume.Here, capacity of the virtual volume will be roughly the same capacityas the total value of the capacity of the respective volumes.Incidentally, as this kind of combination method, there are striping,concatenate (method of linking a plurality of volumes and treating themas a single volume) and so on.

(D) In addition to pattern (C), further storing parity information ormirror data. Here, capacity of the virtual volume will be half of (C)when storing mirror data, or depend on the parity calculation methodwhen storing parity. Reliability of data stored in the virtual volumecan be improved through combination with high-reliability based on RAIDinside the storage apparatus 1500.

Incidentally, regarding every pattern, the storage apparatus identifier(or port identifier) and the volume identifier (information foridentifying volumes in the virtual storage apparatus or controlled byports used in the I/O request, such as LUN (Logical Unit Number),CKD-format CU number, LDEV (Logical DEVice) number, and the like),designated in the I/O request, differ from the original volume.

The virtualization program 6030 is called by the I/O processing program6020 when the data to be subject to staging or destaging corresponds tothe virtual volume, and uses the virtualization information 6070 toexecute the following processing.

(A) Staging: Deciding which data stored in the volume of which storageapparatus 1500 should be copied to the cache memory 1020 based on thecorrespondence of the virtualization volume and the volume of thestorage apparatus 1500, and thereafter copying such data to the cachememory 1020.

(B) Destaging: Deciding which volume of the storage apparatus 1500should be target to copy data in the cache memory 1020 to, based on thecorrespondence of the virtual volume and the volume of the storageapparatus 1500, and thereafter copying such data to the storageapparatus 1500. Incidentally, as the pre-processing thereof, redundantdata based on RAID technology may also be created.

<4.3. Remote Copy Program 6010 and Copy Pair Information 6040>

The copy pair information 6040 possesses the following information foreach copy pair (hereinafter sometimes abbreviated as a “pair”) of thecopy source volume and the copy destination volume of remote copy.Incidentally, in this embodiment, volumes that are the target of highavailability are designated as the copy source volume and the copydestination volume.

(1) Identifier of the virtual storage apparatus 1000 having the copysource volume, and identifier of the volume

(2) Identifier of the virtual storage apparatus 1000 having the copydestination volume, and identifier of the volume

(3) Status of the copy pair (details will be described later)

The remote copy program 6010 is a program for mirroring the data storedin the copy source volume to the copy destination volume, and refers tothe copy pair information 6040 to perform the processing. The processingoverview and pair status of remote copy (in particular synchronousremote copy) are explained below.

<4.3.1. Copy Processing Operation of Synchronous Remote Copy>

As the method of the synchronous remote copy described above, when thevirtual storage apparatus 1000 of the copy source receives a writerequest for writing into the copy source volume from the host 1100, thevirtual storage apparatus 1000 of the copy source sends write data tothe virtual storage apparatus 1000 of the copy destination andthereafter returning a write request completion notice to the host 1100.

When synchronous remote copy is to be executed, the controller 1010 ofthe virtual storage apparatus 1000 manages information referred to as acopy pair status (Simplex, Initial-Copying, Duplex, Suspend andDuplex-Pending), in order to display the status of remote copy betweenthe pair of copy source volume and copy destination volume on amanagement screen 1200 or operate the status of remote copy. FIG. 7shows a status transition diagram relating to the pair status ofsynchronous remote copy. The respective pair statuses are explainedbelow.

<4.3.1.1. Simplex Status>

The Simplex status is a status where copy between the copy source volumeand the copy destination volume configuring a pair has not been started.

<4.3.1.2. Duplex Status>

The Duplex status is a status where synchronous remote copy has beenstarted, the initialization copy described later is complete and thedata contents of the copy source volume and the copy destination volumeconfiguring a pair are the same. In this status, excluding the areasthat are currently being written, data contents of the copy sourcevolume and data contents of the copy destination volume will be thesame. Incidentally, during the Duplex status and in the Duplex-Pendingand Initial-Copying statuses, write requests from the host 1100 to thecopy destination volume are denied.

<4.3.1.3. Initial-Copying Status>

The Initial-Copying status is an intermediate status during thetransition from the Simplex status to the Duplex status, andinitialization copy from the copy source volume to the copy destinationvolume (copy of data already stored in the copy source volume to thecopy destination volume) is performed as required during this period.When initialization copy is complete and processing necessary for makingthe transition to the Duplex status is complete, the pair status becomesa Duplex status.

<4.3.1.4. Suspend Status>

The Suspend status is a status where the contents written into the copysource volume are not reflected in the copy destination volume. In thisstatus, data contents of the copy source volume and the copy destinationvolume configuring a pair are not the same. Triggered by a command fromthe user or the host 1100, the pair status makes a transition fromanother status to the Suspend status. In addition, a case may beconsidered where, when it is no longer possible to perform synchronousremote copy due to a network failure or the like between the virtualstorage apparatuses 1000, the pair status makes an automatic transitionto the Suspend status.

In the following explanation, the latter case; that is, the Suspendstatus caused by a failure will be referred to as a Failure Suspendstatus. Representative examples that cause such Failure Suspend statusare, in addition to a network failure, failures in the copy sourcevolume and the copy destination volume, and failure of the controller1010.

When entering the Suspend status, although the copy source storage 1000receives write data according to a write request and stores it in thecopy source volume when such write request is issued to the copy sourcevolume subsequent to entering the Suspend status, the copy sourcestorage 1000 does not send the write data to the virtual storageapparatus 1000 of the copy destination. Further, the virtual storageapparatus 1000 of the copy source stores the writing location of thewritten write data in the copy source volume as a differential bitmap orthe like.

Incidentally, when a write request is issued to the copy source volumesubsequent to entering the Suspend status, the virtual storage apparatus1000 of the copy destination also performs the foregoing operation.Further, when a setting referred to as “fence” is configured in a pairbefore such pair enters the Failure Suspend status, writing of the copysource volume is denied after the pair status makes a transition to theFailure Suspend status. Incidentally, the virtual storage apparatus 1000of the copy destination may also deny the write request to the copydestination volume during the Failure Suspend status.

<4.3.1.5. Duplex-Pending Status>

The Duplex-Pending status is the intermediate status during thetransition from the Suspend status to the Duplex status. In this status,data copy from the copy source volume to the copy destination volume isexecuted in order to make the data contents of the copy source volumeand the copy destination volume coincide. After the data contents of thecopy source volume and the copy destination volume become identical, thepair status becomes a Duplex status.

Incidentally, data copy during the Duplex-Pending status is executed,via differential copy of copying only the portions that need to beupdated (in other words, the inconsistent data between the copy sourcevolume and the copy destination volume) by using the writing location(for instance, the foregoing differential bitmap or the like) recordedin the virtual storage apparatus 1000 of the copy source or the virtualstorage apparatus 1000 of the copy destination during the Suspendstatus.

Further, although the Initial-Copying status and the Duplex-Pendingstatus were explained above as being separate statuses, these may alsobe combined and displayed as one status on the screen of the managementhost 1200, or subject to transition as one status.

<4.3.1.6. Pair Operation Command>

The pair status makes a transition to another status based on thefollowing commands from the host 1100 or the management host 1200.

(A) Initialization command: When this command is received during theSimplex status, transition is made to the Initial-Copying status.

(B) Resynchronization command: When this command is received during theSuspend status or the Failure Suspend status, transition is made to theDuplex-Pending status.

(C) Partition command: When this command is received during the Duplexstatus, transition is made to the Suspend status.

(D) Copy direction inversion command: When this command is receivedduring the Duplex status, Suspend status or Failure Suspend status,relationship of the copy source and the copy destination is inverted. Inthe case of a Duplex status, the copy direction is also inverted whenthis command is received.

Incidentally, the initialization command is expected to designate thevirtual storage apparatus 1000 of the copy source and the copy sourcevolume, and the virtual storage apparatus 1000 of the copy destinationand the copy destination volume, and the remaining commands merely needto designate identifiers showing the pair relationship since such pairrelationship has already been formed (combination of the virtual storageapparatus 1000 of the copy source and the copy source volume, and thevirtual storage apparatus 1000 of the copy destination and the copydestination volume is also one of such identifiers).

<5. Programs and Information to be Executed by Storage Apparatus 1500>

FIG. 6 illustrates the programs and information to be executed by thestorage apparatus 1500, and the respective programs and informationperform the same operation as the virtual storage apparatus 1000.

<6. Device Relation Table 5001>

FIG. 8 is a diagram showing the information contained in the devicerelation table 5001. The device relation table 5001 manages thefollowing information for each virtual volume (more specifically, foreach identifier corresponding to such volume) in the host 1100 providedby the I/O path manager 5000.

(A) Identifiers of the virtual volumes in the host 1100

(B) Related volume identifier list: Identifiers of volumes of thestorage apparatus 1500 that may become the entity of virtual volumes inthe host 1100 are included. Incidentally, as said individualidentifiers, the identifiers allocated by the HBA device drivers 5010 asthe lower layer of the I/O path manager 5000 are used. In thisembodiment, identifiers of volumes in the primary virtual storageapparatus 1000 (1000L) and volumes in the secondary virtual storageapparatus 1000 (1000R) are listed (if a normal status).

(C) Primary volume: Shows which volume listed at (B) is a primary.

(D) Failure status

(E) Pair status

Incidentally, since the identifiers of (A) and the identifiers of (B)are handled the same from the perspective of the file system 5020,overlap of the identifiers of (A) and (B) is not allowed. Further, sinceoverlap is also not allowed in the case of combining (A) and (B), theI/O path manager 5000 needs to create the identifiers of (A) whilegiving consideration to this point.

<7. Initialization Processing>

FIG. 9 is a flowchart illustrating the initialization processing of theI/O path manager 5000. This initialization processing is now explainedwith reference to the flowchart. Incidentally, although there are casesbelow where the processing subject of various processes is explained asthe “I/O path manager 5000,” in reality, it goes without saying that theprocessor 1101 (FIG. 1) of the host 1100 executes the correspondingprocessing based on a program called the “I/O path manager 5000.”

(S9001) The I/O path manager 5000 receives an initialization commandcontaining the following information from the user of the managementhost 1200 or the host 1100. Incidentally, as the initializationprocessing of a duplex system, this is also referred to as an HA (HighAvailability) initialization command.

(A) Primary virtual storage apparatus 1000 and its volumes

(B) Secondary virtual storage apparatus 1000 and its volumes

(S9002) The I/O path manager 5000 communicates with both virtual storageapparatuses 1000 commanded at S9001 and acquires the existence ofvolumes and the capacity thereof.

(S9003) The I/O path manager 5000 confirms that volumes commanded atS9001 exist and are of the same capacity. When this cannot be confirmed,the I/O path manager 5000 returns an error to the command source.

(S9004) The I/O path manager 5000 sends a remote copy initializationcommand to one or both virtual storage apparatuses 1000. Thisinitialization command is commanded with the primary volume as the copysource volume and the secondary volume as the copy destination volume.Based on this command, the virtual storage apparatus 1000 starts remotecopy.

(S9005) The I/O path manager 5000 registers the following information inthe device relation table 5001, and thereafter returns an initializationstart reply to the source of the initialization command.

(A) Identifiers of the virtual volumes in the host 1100 (=values createdby the I/O path manager 5000)

(B) Related volume identifier list (=two identifiers corresponding tothe virtual storage apparatus 1000 and the volume designated at S9001(both the primary system and secondary system)).

(C) Identifier of the primary volume (=primary volume designated atS9001)

(D) Failure status (=secondary system in preparation)

(E) Pair status (=Initial-Copying)

(S9006) The I/O path manager 5000 monitors the pair status of remotecopy, and updates the device relation table 50001 to the followinginformation upon transition to the Duplex status.

(D) Failure status (=normal status)

(E) Pair status (=Duplex)

As a result of the foregoing processing, the I/O path manager 5000 isable to start the preparation for high availability including thesetting of remote copy according to the user's command. Incidentally, inreality, since the I/O path manager 5000 is able to provide the virtualvolume in the host 1100 immediately after S9005, users who wish to makeaccess in a file format is able to start file I/O by issuing a mountcommand to the volume. Further, as a different method, the I/O pathmanager 5000 may define the virtual volume in the host 1100corresponding to the volume to realize high availability before thesetting of remote copy, and the file system 5020 may also start theforegoing processing from a state of mounting the volume by the userdesignating a volume to become a secondary system.

<8. Write Request Processing Flow>

FIG. 10 is a diagram showing the processing flow when the I/O pathmanager 5000 receives a write request from the file system 5020.

(S10001) From the file system 5020, the I/O path manager 5000 is called(or receives a message of) a write request function including theidentifier of the virtual volume in the host 1100 to become the writedestination, write location of the volume, and the write length.

(S10002) The I/O path manager 5000 confirms the failure status of thevirtual volume and, if it is a remote copy failed status, transfers thecontrol to the dual write processing at S10020, and otherwise executesS10003.

(S10003) The I/O path manager 5000 issues a write request to the primaryvolume. Incidentally, issuance of the write request is actually realizedby calling the HBA device drive 5010 of the lower layer.

(S10004) The I/O path manager 5000 confirms the reply of the writerequest, returns a completion reply to the file system 5020 if it is anormal end or transfers the control to the dual write processing atS10020 if it is a remote copy failure or transfers the control to theswitch processing at S10010 if it is a no reply or in other cases.

Incidentally, the dual write processing at S10020 is executed at thefollowing steps.

(S10021) If the writing into the primary or secondary volume is denieddue to the setting of remote copy, the I/O path manager 5000 cancelsthis setting.

(S10022) The I/O path manager 5000 issues a write request to the primaryvolume.

(S10023) The I/O path manager 5000 issues a write request to thesecondary volume. The I/O path manager 5000 waits for the arrival of awrite request reply from both the primary system and secondary system,and returns a completion reply to the file system 5020.

<8.1. Flow of Switch Processing>

The processing realized by the switch processing is further explained.

(S10011) The I/O path manager 5000 foremost confirms whether thesecondary volume is available by referring to the failure status of thedevice relation table 5001, and returns an error reply to the filesystem 5020 if it determines that the secondary volume is unavailable,or executes S10012 if the secondary volume is available. Incidentally, astatus where there is no secondary system (when the secondary virtualstorage apparatus 1000 is not functioning due to a failure, or in a caseof a volume in which the secondary virtual storage apparatus 1000 is notset to begin with), and the status of initialization in preparationdescribed above may consider the status of unavailable.

(S10012) The I/O path manager 5000 issues a remote copy stop command tothe secondary virtual storage apparatus 1000 and, after confirming thatthe copy status entered the Suspend status, issues a copy directioninversion command.

(S10013) The I/O path manager 5000 issues a remote copyresynchronization command to the secondary virtual storage apparatus1000. Incidentally, there is no need to wait until the resynchronizationis actually complete and the pair status enters the Duplex status.

(S10014) The I/O path manager 5000 updates the primary volume identifierof the device relation table 5001 to a volume identifier that was asecondary system theretofore, and switches the primary system and thesecondary system. Then, the I/O path manager 5000 sends a write requestto the new primary volume through the HBA device driver 5010.

(S10015) The I/O path manager 5000 confirms the reply of the writerequest, returns a completion reply to the file system 5020 if it is anormal end or returns an error reply if it is an error, and ends theprocessing.

<8.1.1. Countermeasures Against Write Request Failure During Dual WriteProcessing>

When the write request to the primary volume at S10022 ends in a failureduring the dual write processing at S10020, control may be transferredto the switch processing at S10010. Further, when the write request tothe secondary volume at S10023 ends in a failure, the failure status ofthe device relation table 5001 is changed to ‘no secondary system,’ andwriting is thereby completed.

Further, since the pair status is a Failure Suspend status during thedual write processing, a write location is indicated in the volume ofthe virtual storage apparatus 1000 based on a differential bitmap ofremote copy. Nevertheless, since the write data written in both volumesbased on the dual write processing are the same, it is desirable toavoid recording in the differential bitmap while the dual writeprocessing is being conducted normally, and to copy only thedifferential data during the resynchronization processing after recoveryof the communication failure. As a solution for the above, while thedual write processing is being conducted normally, a case may beconsidered of periodically and repeatedly clearing the differentialbitmap of the volume of both the primary and secondary virtual storageapparatuses 1000. With this method, there is no need to issue a clearcommand for each write request, and it is possible to avoid the copy ofall areas of the target volume during the resynchronization of remotecopy. This is because, although the write request of the dual writeafter the time of the nearest clearing process and the write request ofthe dual write during the failure of the dual write will be recorded asa write location in the differential bitmap, there will be no datainconsistency or copy omission area. Because, even when the data arearecorded during the dual write is copied with resynchronization, thedata contents of the copy destination will not change.

Incidentally, in the foregoing solution, processing of the write requestmay be temporarily stopped in order to clear the differential bitmap ofboth the primary and secondary system. As a method of stopping theprocessing, considered may be a method of the I/O path manager 5000 nottransferring the write request received from the file system 5020 to thevirtual storage apparatus 1000 until both differential bitmaps arecleared, or a method of pending the write request processing in theprimary virtual storage apparatus 1000 until both differential bitmapsare cleared.

As a second solution, there is a method of allocating two differentialbitmaps respectively to the primary and secondary volumes. Theprocessing contents thereof are shown below.

(Initial status) The primary and secondary virtual storage apparatuses1000 respectively record the location of the write request on one sideof the two differential bitmaps. Thus, both virtual storage apparatuses1000 will retain and manage information concerning an active side (thisside refers to the side recording the write location when the writerequest arrives, and the other side of the differential bitmap isreferred to as an inactive side). Further, it is desirable that there isnothing recorded on the inactive side of the differential bitmap.

(Step 1) The primary virtual storage apparatus 1000 switches thedifferential bitmap to become the recording destination of the locationof the write request and the subsequent write requests are recorded inthe switched differential bitmap by updating the management informationof the active side to an alternative differential bitmap that was aninactive side. The secondary virtual storage apparatus 1000 is similarlyswitched. Incidentally, the trigger for starting the switch processingis given from the I/O path manager 5000 to both virtual storageapparatuses 1000. Incidentally, the switch processing of the primarysystem and secondary system may be executed in any order, or may beexecuted in parallel.

(Step 2) The I/O path manager 5000 issues a differential bitmap clearcommand to both virtual storage apparatuses 1000 upon waiting for aswitch completion reply from both virtual storage apparatuses 1000. Thevirtual storage apparatus 1000 that received the clear command clearsthe write location of the differential bitmap that is an inactive side,and returns a reply to the I/O path manager 5000. Similar to the switchprocessing, the clear processing of the primary system and secondarysystem may be executed in any order, or may be executed in parallel.

(Step 3) The I/O path manager 5000 waits for a clear completion replyfrom the both virtual storage apparatuses 1000, and re-executes theprocess from Step 1 after the lapse of a certain period of time.

In the case of this solution, with the resynchronization processingafter recovery of the communication failure, the area to performdifferential copy can be decided during the Duplex-Pending status bycalculating the logical sum of four bitmaps of the primary system andsecondary system. Further, although there are many bitmaps in thismethod, there is no need to pend the write request.

The following third solution is a modified example of the foregoingsecond solution.

(Initial status) The primary and secondary virtual storage apparatuses1000 respectively record the location of the write request on both sideof the differential bitmaps. Thus, both virtual storage apparatuses 1000will retain and manage information concerning the differential bitmapside that was previously cleared.

(Step 1) The I/O path manager 5000 issues a differential bitmap clearcommand to both virtual storage apparatuses 1000. The virtual storageapparatus 1000 that received the clear command clears the write locationof the alternative differential bitmap that is not the different bitmapthat was cleared previously, and returns a reply to the I/O path manager5000.

(Step 3) The I/O path manager 5000 waits for a clear completion replyfrom the both virtual storage apparatuses 1000, and re-executes theprocess from Step 1 after the lapse of a certain period of time.

<9. Read Request Processing Flow>

FIG. 11 is a flowchart showing the processing contents when the I/O pathmanager 5000 receives a read request from the file system 5020.

(S11001) From the file system 5020, the I/O path manager 5000 is called(or receives a message of) a read request function including theidentifier of the virtual volume in the host 1100 to become the readdestination, read location of the volume, and the read length.

(S11002) The I/O path manager 5000 confirms the failure status of thevirtual volume, executes S11021 if it is a normal status and the I/Oload against the primary volume is high (for instance, when a given IOPSis exceeded or a given bandwidth is exceeded) or otherwise executesS11003 (no secondary system, secondary system in preparation, normalstatus, etc.).

(S11003) The I/O path manager 5000 issues a read request to the primaryvolume.

(S11004) The I/O path manager 5000 confirms the reply of the readrequest, returns a completion reply to the file system 5020 if it is anormal end or transfers the control to the switch processing at S11010in other cases.

(S11021) The I/O path manager 5000 issues a read request to thesecondary volume.

(S11022) The I/O path manager 5000 confirms the reply of the readrequest, returns a completion reply to the file system 5020 if it is anormal end or executes S11023 in other cases.

(S11023) The I/O path manager 5000 updates a failure status of thedevice relation table 5001 to ‘no secondary system,’ and executesS11003.

<9.1. Flow of Switch Processing>

The processing realized by the switch processing is further explained.

(S11011) The I/O path manager 5000 foremost confirms whether thesecondary volume is available by referring to the failure status of thedevice relation table 5001, and returns an error reply to the filesystem 5020 if it determines that the secondary volume is unavailable orexecutes S11012 if the secondary volume is available. Incidentally, as astatus of being determined as being unavailable, considered may be astatus where there is no secondary system (when the secondary virtualstorage apparatus 1000 is not functioning due to a failure, or in a caseof a volume in which the secondary virtual storage apparatus 1000 is notset to begin with), and the status of initialization in preparationdescribed above.

(S10012) The I/O path manager 5000 issues a remote copy stop command tothe secondary virtual storage apparatus 1000 and, after confirming thatthe copy status entered the Suspend status, issues a copy directioninversion command.

(S10013) The I/O path manager 5000 issues a remote copyresynchronization command to the secondary virtual storage apparatus1000. Incidentally, there is no need to wait until the resynchronizationis actually complete and the pair status enters the Duplex status.

(S10014) The I/O path manager 5000 updates the primary volume identifierof the device relation table 5001 to a volume identifier that was asecondary system theretofore, and switches the primary system and thesecondary system. Then, the I/O path manager 5000 sends a read requestto the new primary volume through the HBA device driver 5010.

(S10015) The I/O path manager 5000 confirms the reply of the readrequest, returns a completion reply to the file system 5020 if it is anormal end or returns an error reply if it is an error and ends theprocessing.

<10. Failure Countermeasure Processing Flow>

In this section, the flow of processing from the time the I/O pathmanager 5000 detects a failure until the recovery is complete isexplained. Incidentally, this processing is periodically executed in thebackground.

<10.1. Network Failure between Virtual Storage Apparatuses 1000>

(Step 1) The I/O path manager 5000 monitors the pair status of remotecopy and detects the occurrence of some kind of failure by discovering aFailure Suspend status.

(Step 2) The I/O path manager 5000 issues a remote copy stop command tothe secondary virtual storage apparatus 1000, inverts the copy directionafter confirming that the copy status entered a Suspend status, andinquires the status to the respective virtual storage apparatuses 1000.Then the I/O path manager 5000 updates the failure status of the devicerelation table 5001 to ‘remote copy failure’ after confirming that nofailure has occurred to the self virtual storage apparatus 1000 and thatthe cause is a network failure. Incidentally, this processing may alsoutilize the work result of the work performed by the storageadministrator.

(Step 3) Wait until the network recovers.

(Step 4) The I/O path manager 5000 issues a pair resynchronizationcommand to the primary virtual storage apparatus 1000.

(Step 5) The I/O path manager 5000 updates the failure status of thedevice relation table 5001 to ‘secondary system in preparation.’

(Step 6) The I/O path manager 5000 waits for the pair status to become aDuplex status, and thereafter updates the failure status of the devicerelation table 5001 to ‘normal status.’

<10.2. Failure and Shutdown of Primary Virtual Storage Apparatus 1000>

(Step 1) The I/O path manager 5000 detects the occurrence of a failureby monitoring the status of the primary virtual storage apparatus 1000.

(Step 2) The I/O path manager 5000 switches the subsequent I/O requestdestination to the secondary virtual storage apparatus 1000 by changingthe identifier of the primary volume of the device relation table 5001to the identifier of the secondary volume, and further updates thefailure status to ‘no secondary system.’

(Step 3) The I/O path manager 5000 waits until the old primary(currently secondary switched at Step 2) virtual storage apparatus 1000recovers.

(Step 4) The I/O path manager 5000 issues a pair resynchronizationcommand or initialization command to the primary virtual storageapparatus 1000.

(Step 5) The I/O path manager 5000 updates the failure status of thedevice relation table 5001 to ‘secondary system in preparation.’

(Step 6) The I/O path manager 5000 waits for the pair status to become aDuplex status, and then updates the failure status of the devicerelation table 5001 to ‘normal status.’

<10.3. Failure and Shutdown of Secondary Virtual Storage Apparatus 1000>

(Step 1) The I/O path manager 5000 detects the occurrence of a failureby monitoring the status of the secondary virtual storage apparatus1000.

(Step 2) The I/O path manager 5000 updates the failure status of thedevice relation table 5001 to ‘no secondary system.’

(Step 3) The I/O path manager 5000 waits until the secondary virtualstorage apparatus 1000 recovers.

(Step 4) The I/O path manager 5000 issues a pair resynchronizationcommand or initialization command to the primary virtual storageapparatus 1000.

(Step 5) The I/O path manager 5000 updates the failure status of thedevice relation table 5001 to ‘secondary system in preparation.’

(Step 6) The I/O path manager 5000 waits for the pair status to become aDuplex status, and then updates the failure status of the devicerelation table 5001 to ‘normal status.’

<11. Alternative Initialization Method>

In the foregoing explanation, although remote copy was configured to thevirtual storage apparatus 1000 according to an initialization requestissued from the I/O path manager 5000, the opposite method describedbelow can also be considered.

(Step 1) The management host 1200 starts remote copy by issuing a remotecopy pair initialization command to the virtual storage apparatus 1000.

(Step 2) The I/O path manager 5000 receives a scanning request.

(Step 3) The I/O path manager 5000 acquires the configuration of remotecopy in the respective volumes through the HBA device driver 5010(status of remote copy configuration, whether it is a copy source or acopy destination, the virtual storage apparatus 1000 to become the otherpair and its volume). Incidentally, as the foregoing acquisition method,a SCSI command can be used in the I/O network, or information can beacquired using other communication networks.

(Step 4) The I/O path manager 5000 creates a device relation table 5001based on the information acquired at the previous step, and starts theprocessing described above. Incidentally, creation examples of thedevice relation table 5001 are shown below.

(A) Identifier of the virtual volume in the host 1100=value created bythe I/O path manager 5000

(B) Related volume identifier list=identifiers of the copy source volumeand the copy destination volume of remote copy

(C) Primary volume=copy source volume remote copy

(D) Failure status=‘normal status’ when the pair status acquired fromthe virtual storage apparatus 1000 is a Duplex status, ‘secondary systemin preparation’ when it is an Initial-Copying status or a Duplex-Pendingstatus, ‘remote copy failure’ when it is a Suspend status or a FailureSuspend status

(E) Pair status=pair status acquired from the virtual storage apparatus1000

High availability is realized in this embodiment based on the operationof the hardware and programs described above. Incidentally, ascountermeasures to be taken when much time is required for the switchprocessing illustrated in FIG. 10 and FIG. 11, a part of the foregoingswitch processing can be executed as preliminary processing when itbecomes necessary for the I/O path manager 5000 to re-send the I/Orequest. Here, the preliminarily performed switch processing can berestored if the re-sent I/O request is returned with a normal reply, andthe remaining portions of the foregoing switch processing can beexecuted if the re-sent I/O request is returned with error reply, orthere is no reply. Further, in this embodiment, all volumes may bevirtualized with the virtual storage apparatus 1000, the entity may be avirtual volume in the storage apparatus 1500, and the virtual storageapparatus 1000 may be an apparatus dedicated to virtualization, orcontrarily a constitution where the entity of all volumes is inside thevirtual storage apparatus 1000 may be adopted. Moreover, in addition tothe capacity, various other attributes may be configured to the volumesprovided by the virtual storage apparatus 1000 (for instance, anemulation type or a volume identification number acquirable with anInquiry command defined based on a SCSI standard).

Such attribute information and attribute change are also transferredfrom the primary virtual storage apparatus to the secondary virtualstorage apparatus based on remote copy, and these may also be managed inboth virtual storage apparatuses.

<12. Alternative Read/Write Processing>

In the write/read processing illustrated in FIG. 10 and FIG. 11, the I/Opath manager 5000 specifically transfers the operation of remote copy tothe virtual storage apparatus 1000. Nevertheless, since the operation ofremote copy may differ for each vendor of the virtual storage apparatus1000, there are cases when it would be more preferable not to includesuch operation in the write processing and read processing of the I/Opath manager 5000. FIG. 25 to FIG. 27 show the processing contents ofsuch a form. Incidentally, although there are cases below where theprocessing subject of various processes is explained as the “virtualstorage apparatus 1000,” in reality, it goes without saying that theprocessor 1101 (FIG. 1) of the virtual storage apparatus 1000 executesthe corresponding processing based on programs stored in the memory 1012(FIG. 1).

<12.1. Write Processing of I/O Path Manager>

FIG. 25 is a flowchart showing the general processing contents of FIG.10 to be executed by the I/O path manager 5000. FIG. 25 differs from theFIG. 10 in the following points.

(Difference 1) The operation of remote copy at steps S10012, S10013 andS10021 is skipped.

(Difference 2) The routine does not reach step S10020 of the flow duringremote copy failure. Nevertheless, these differences only occur when itis not possible to identify an error message signifying remote copyfailure in normal read/write processing.

FIG. 27 is a diagram showing the operation of remote copy to beperformed when the virtual storage apparatus 1000 receives a writerequest.

(S27001) The virtual storage apparatus 1000 receives a write request.

(S27002) The virtual storage apparatus 1000 determines whether thetarget volume of the write request is related to remote copy, andexecutes S27003 when it is unrelated, and executes S27004 when it isrelated.

(S27003) The virtual storage apparatus 1000 performs normal writeprocessing, returns a reply to the host 1100 and ends this processing.

(S27004) The virtual storage apparatus 1000 determines the remote copyattribute of the target volume of the write request, and executes S27005when it is a copy source attribute, and executes S27011 when it is acopy destination attribute.

(S27005) The virtual storage apparatus 1000 executes synchronous remotecopy processing, transfers write data to the secondary storage, andwaits for a reply.

(S27006) The virtual storage apparatus 1000 determines whether the copywas successful, and executes S27008 if the copy was successful, andexecutes S27007 is the copy was unsuccessful.

(S27007) The virtual storage apparatus 1000 changes the status of theremote copy pair in which the target volume will become the copy sourceto a Failure Suspend status. However, writing to this volume is notprohibited.

(S27008) The virtual storage apparatus 1000 performs normal writeprocessing, returns a reply to the host 1100, and ends this processing.

(S27011) The virtual storage apparatus 1000 stops remote copy, andinverts the relationship of the copy source and the copy destination.

(S27012) The virtual storage apparatus 1000 starts the resynchronizationprocessing.

(S27013) The virtual storage apparatus 1000 performs normal writeprocessing, returns a reply to the host 1100, and then ends thisprocessing.

Incidentally, it is not necessary to wait until the resynchronizationprocessing at S27012 is complete. This is because the virtual storageapparatus 1000 executing S27012 is a secondary system, the primaryvirtual storage apparatus 1000 is not necessarily operating normally,and much time may be required until the resynchronization processing iscomplete. Incidentally, the foregoing case is the same in that it can berecovered with the processing described in <10. Failure MeasureProcessing Flow>.

<12.3. Read Processing of I/O Path Manager>

FIG. 26 is a flowchart showing the general processing contents of FIG.11 to be executed by the I/O path manager 5000. FIG. 26 differs from theFIG. 11 in the following point.

(Difference 1) The operation of remote copy at steps S11012 and S11013is skipped.

Incidentally, although in FIG. 11 the direction of remote copy wasinverted according to the read processing, the remote copy direction isnot inverted in this processing. This is because, in addition to caseswhere the primary virtual storage apparatus 1000 will not return a replyto the read request to the secondary virtual storage apparatus 1000(including cases caused by a communication failure between hosts=virtualstorage apparatuses), there are cases where this is caused by the excessload of the primary virtual storage apparatus 1000. Thus, if thesecondary virtual storage apparatus 1000 performs the pair inversion ofremote copy triggered by the read request to the copy destinationvolume, the pair will be inverted with the read request that justhappened to be issued to the secondary virtual storage apparatus 1000,and the pair will be inverted once again with the subsequent readrequest, and the read performance will deteriorate as a result.

Nevertheless, when the execution of S11021 is inhibited, the virtualstorage apparatus 1000 may perform pair inversion of remote copy byperforming the following processing upon read processing.

(Step 1) The virtual storage apparatus 1000 receives a read request.

(Step 2) The virtual storage apparatus 1000 performs normal readprocessing.

(Step 3) The virtual storage apparatus 1000 determines whether theread-target volume is the copy destination volume of remote copy, andexecutes subsequent Step 4 if so, and ends this processing if not.

(Step 4) The virtual storage apparatus 1000 stops remote copy, andinverts the relationship of the copy source and the copy destination.

(2) Second Embodiment

The second embodiment is now explained with reference to FIG. 12. Thesecond embodiment differs from the first embodiment in that the storageapparatus 1500L is coupled to a plurality of virtual storage apparatuses1000L, 1000R, and these virtual storage apparatuses 1000L, 1000R sharethe volumes in the storage apparatus 1500L to enable the continuation ofservice at a lower cost than the first embodiment even when one of thevirtual storage apparatuses 1000L, 1000R shuts down.

Nevertheless, since the virtual storage apparatuses 1000L, 1000R includecache memories 1020L, 1020R, in preparation for a case when the primaryvirtual storage apparatus 1000L shuts down due to a disaster immediatelyafter write data is written into the virtualization volume, it isnecessary to also store the write data into the cache memory 1020R ofthe secondary virtual storage apparatus 1000R, and the destaging andstaging of both virtual storage apparatuses 1000L, 1000R must be devisedaccordingly.

A write request in a normal status is processed according to thefollowing steps.

(Step 1) The primary virtual storage apparatus 1000L that received awrite request from the host 1100 determines whether the write request isaddressed to the volume 3000LA corresponding to the HDD 1030 inside thevirtual storage apparatus 1000L, addressed to the virtualization volume(hereinafter referred to as the “shared virtualization volume”) 3000LBprovided by both virtual storage apparatuses 1000L, 1000R by sharing thevolume 3500L of the storage apparatus 1500L, or addressed to the normalvirtualization volume. Incidentally, processing other than the sharedvirtualization volume 3000LB is the same as the processing of the firstembodiment.

(Step 2) The primary virtual storage apparatus 1000L stores the writedata in its internal cache memory 1020L, stores the write data in thecache memory 1020R of the secondary virtual storage apparatus 1000Rbased on a remote copy program, and thereafter returns a normal reply tothe host 1100.

(Step 3) The caching algorithm of the primary virtual storage apparatus1000L decides the data in the cache memory 1020L to be destaged, anddestages the data to the volume of the storage apparatus 1500L.

(Step 4) After destaging is complete, the primary virtual storageapparatus 1000L commands the secondary virtual storage apparatus 1000Rto discard the address of data in the destaged cache memory 1020L.Incidentally, the secondary virtual storage apparatus 1000R thatreceived the command discards the target data from the cache memory1020R.

Incidentally, in this constitution, when switching of the I/O request isconducted to the secondary virtual storage apparatus 1000R in a statewhere the network between the virtual storage apparatuses 1000L, 1000Ris disconnected, there are cases where the virtual storage apparatuses1000L, 1000R will both autonomously perform destaging as primarysystems. In order to avoid this kind of situation, when both virtualstorage apparatuses 1000L, 1000R are to perform processing as primarysystems, they may foremost perform exclusion control using a functionsuch as SCSI Reserve or the like to the volume 3500L shared in thestorage apparatus 1500L. Further, as another method, caching of virtualstorage apparatus 1000L may be invalidated regarding the sharedvirtualization volume 3000LB, and, in such a case, when the accessauthority of the shared virtual volume 3000LB is changed to a read-onlyaccess authority, caching may be validated according to such change.

(3) Third Embodiment

The third embodiment is now explained with reference to FIG. 13. In thisembodiment, the information system described in the foregoingembodiments is separately prepared at a remote site (backup site) thatis different from the production site to perform remote copy, and theservice can be resumed at the backup site when the production site issubject to a disaster.

Incidentally, in the following explanation, there are cases where theforegoing “virtual storage apparatus” is referred to as a storageapparatus, the “copy source volume” as a primary volume, the “copydestination volume” as a secondary volume, the “primary system” as anactive side, and the “secondary system” as a standby side. Further, theinformation systems of the production site and the backup site may becollectively referred to as a remote copy system.

<1. Constitution of Remote Copy System>

In this embodiment, each site is constituted of hosts 13010, 13020 and aplurality of storage subsystems 13001, 13002, 13003, 13004. At theproduction site, the storage subsystems 13001, 13002 jointly adopt thehigh availability constitution described above. Moreover, at the backupsite also, the storage subsystems 13003, 13004 jointly adopt the highavailability constitution.

Further, in this embodiment, synchronous or asynchronous remote copy isperformed from the active-side storage subsystem (with a copy sourcevolume) 13001 of the production site to the active-side storagesubsystem (with a copy destination volume) 13003 of the backup site.When the production site is subject to a disaster, the host 1310 of thebackup site issues an I/O request to active side of the storagesubsystems 13003, 13004 of a high availability constitution, and there-booted application 2010 thereby resumes the processing.

Incidentally, as described above, a storage subsystem refers to bothconcepts including a constitution that does not use the virtualizationfunction of the virtual storage apparatus 1000 (FIG. 1), as well as to aconstitution where the virtual storage apparatus 1000 provides avirtualization volume using the virtualization function based on acombination of the virtual storage apparatus 1000 and the storageapparatus 1500 (FIG. 1). Further, in this embodiment, each storagesubsystem 13001, 13002, 13003, 13004 may adopt separate internalconstitutions (for instance, configuring only the storage subsystem13001 with the virtual storage apparatus 1000 without using thevirtualization function, or sharing the storage apparatus 1500 (FIG. 1)with the storage subsystems 13003 and 13004 of the backup site, but notsharing the same on the production site side).

Incidentally, although there are cases below where the processingsubject of various processes is explained as the “storage subsystem,” inreality, it goes without saying that the processor of the storagesubsystem executes the corresponding processing based on programs storedin the memory of the storage subsystem.

<2. Processing>

When the application 2010 of the host 1301 of the production site issuesa write request, the OS determines the active-side storage subsystem inthe production site, and transfers the write request thereto.Incidentally, the storage subsystem 13001 corresponds to this in FIG.13.

The active-side storage subsystem 13001 of the production site transferswrite data to the standby-side storage subsystem (13002 corresponds tothis in FIG. 13) in the production site based on synchronous remotecopy. Further, the active-side storage subsystem 13001 transfers writedata to the active-side storage subsystem (13003 corresponds to this inFIG. 13) of the backup site as synchronous or asynchronous remote copy(since only the active side processes the write request in the highavailability constitution in this embodiment, remote copy is alsosimilarly processed on the active side). The active-side storagesubsystem 13003 in the backup site that received the write datatransfers the received write data to the standby-side storage subsystem13004 in the site based on synchronous remote copy.

Thus, the storage subsystems 13001, 13002 of the production site arekeeping track of the active-side storage subsystem of the backup site,and the storage subsystems 13003, 13004 of the backup site are alsokeeping track of the active storage subsystem (storage subsystem 1301)of the production site so that they will not accept remote copy from anunexpected storage subsystem.

As a result of the foregoing processing, high availability is realizedin both the production site and the backup site. However, the backupsite may be of a constitution that does not adopt the high availabilityconstitution for reduction of costs.

<3. Asynchronous Remote Copy>

Unlike with synchronous remote copy described above, asynchronous remotecopy does not transfer write data at the time a write request arrivesfrom the host 13010, but rather transfers such write data after therequest completion reply (to put it differently, asynchronous remotecopy transfers write data in a timing independent from the request replyto the host 13010). Thus, with asynchronous remote copy, it is possibleto perform remote copy without deteriorating the response time of thewrite request even when the communication delay is significant becausethe distance between the sites is long. Nevertheless, with asynchronousremote copy, it is necessary to buffer write data in the storagesubsystem 13001 on the side of the production site. The followingmethods for buffering write data may be considered.

(1) The storage subsystem 13001 of the production site creates a journalcontaining write data to the copy source volume and sequence informationof such write data, stores this in its own cache memory or a dedicatedvolume, transfers this journal to the storage subsystem 13003 of thebackup site, and the storage subsystem 13003 of the backup site storeswrite data in the copy destination volume by referring to the sequenceinformation of the journal. Thereby, when the production site is subjectto a disaster, it is possible to provide data with a protected writesequence (more specifically, write data with dependence on the side ofthe backup site.

(2) The storage subsystem 13001 of the production site groups the datawritten into the copy source volume every given period and stores suchgroup in its own cache memory or a dedicated volume, transfers thisasynchronously to the storage subsystem 13003 of the backup site, andstores data in group units in the copy destination volume of the storagesubsystem 13003 of the backup site.

Thus, unless the write data to be buffered for asynchronous remote copyis also retained in the standby-side storage subsystem 13002, it willnot be possible to succeed the asynchronous remote copy when theactive-side storage subsystem 13001 shuts down. Thus, the active-sidestorage subsystem 13001 of the production site conveys, in addition towrite data, information of the copy destination volume, foregoingsequence information or timing of performing the grouping process to thestandby-side storage subsystem 13002, and the standby-side storagesubsystem 13002 creates buffering data for asynchronous remote copy asthe same as the active side according to such information.

Incidentally, since the storage subsystem 13003 of the backup sitebuffers the write data received from the production site withoutimmediately storing it in the copy destination volume, the standby sideneeds to similarly create buffering data according to commands from theactive side as with the production site, and store the write data in thecopy destination volume at the same timing.

(4) Fourth Embodiment

The fourth embodiment is now explained with reference to FIG. 14. Thisembodiment explains the constitution of an interface (function I/F) forcontrolling the function provided by a storage apparatus in aninformation system constituted redundantly by two storage apparatusesusing synchronous remote copy described above.

Incidentally, from this embodiment to the fourteenth embodiment, thecomponents referred to as virtual storage apparatuses 1000L, 1000R andstorage apparatuses 1500L, 1500R heretofore are respectively referred toas storage apparatuses 15000A, 15000B and external storage apparatuses16000A, 16000B. Further, although there are cases below where theprocessing subject of various processes is explained as the “storageapparatuses 15000A, 15000B” or the “external storage apparatuses 16000A,16000B,” in reality, it goes without saying that the processors (notshown) of the storage apparatuses 15000A, 15000B or the processors (notshown) of the external storage apparatuses 16000A, 16000B execute thecorresponding processing based on programs stored in the memory of thestorage apparatuses 15000A, 15000B or the external storage apparatuses16000A, 16000B.

This embodiment illustrates an example where, after a function controlrequest from the host 14000 is sent to the storage apparatus 15000A, thestorage apparatus 15000A transfers the function control request to thestorage apparatus 15000B, and both storage apparatuses 15000A, 15000Binterpreting and executing such function control request.

The command device 15002A and the command device 15002B are respectivelythe logical volumes provided by the storage apparatus 15000A and thestorage apparatus 15000B, and act as an interface with the host 1400that controls the function. Incidentally, in this embodiment, it ishypothesized that the command device 15002A is the active side.

Further, based on synchronous remote copy, contents of the commanddevice 15002A and contents of the command device 15002B will constantlycoincide. The command device 15002A and the command device 15002B areprovided to the function management program 14003 as one volume 14004based on the path management function (corresponds to the functionprovided by the I/O path manager 5000 (FIG. 1)) provided by theoperating system 14001.

The logical volume 15001A and the logical volume 15001B are respectivelylogical volumes provided by the storage apparatus 15000A and the storageapparatus 15000B, and logical volumes that are subject to the functioncontrol. Incidentally, in this embodiment, the logical volume 15001A ishypothesized as the active side.

Further, based on synchronous remote copy, contents of the logicalvolume 15001A and contents of the logical volume 15001B will constantlycoincide. The logical volume 15001A and the logical volume 15001B areprovided to the application program 14002 as one volume 14006 based onthe path management function provided by the operating system 14001.

Incidentally, there may be a plurality of logical volumes to be subjectto the function control described above.

The function control request processing unit 14005 of the functionmanagement program 14003 receives a function control request from theuser or another program in the host 14000 or a program in a separatehost (management host or the like) from the host 14000. The functioncontrol request processing unit 14005 that received the function controlrequest reads/writes contents of the control request to the volume 14004from and into the volume 14004. In this embodiment, since the commanddevice 15002A is an active side, the write/read command is issued to thecommand device 15002A.

Writing into the command device 15002A is used to boot the functioncontrol, and reading from the command device 15002A is used to obtainthe output value of the result of the function control.

The control request received by the function control request processingunit 14005 contains information (also referred to as “apparatusinformation”) for uniquely identifying the control-target storageapparatuses 15000A, 15000B, information (also referred to as “volumeinformation”) for uniquely identifying the control-target logicalvolumes 15001A, 150001B, and information incidental to the functioncontrol.

The control I/F processing unit 15003A of the storage apparatus 15000Adetects that a control request has been written into the command device15002A. The control I/F processing unit 15003A determines whether theapparatus information of the control request coincides with theself-storage apparatus (storage apparatus 15000A) (determination 100).In this embodiment, since the command device 15002A is the active side,the determination result will be “Match.” In the case of a match, thecontrol I/F processing unit 15003A calls the function processing unit15004A to execute prescribed function control to the logical volume15001A corresponding to the volume information. As a specific example,there is a referral operation of a pair status of a certain logical copyfunction (described later) as one function provided by the storageapparatus 15000A. When this operation is called to the logical volume15001A, the function processing unit 15004A refers to the managementinformation of the local copy function, and, after acquiring the pairstatus, sends the pair status to the function control request processingunit 14005 via the control I/F processing unit 15003A, the commanddevice 15002A and the volume 14004.

Meanwhile, although the control I/F processing unit 15003B of thestorage apparatus 15000B performs similar processing, in thisembodiment, since the command device 15002B is the standby side, theresult of determination 100 will be “No Match.” In this case, thecontrol I/F processing unit 15003B refers to the pair managementinformation of synchronous remote copy, and specifies the logical volume(corresponds to the logical volume 15001B) in the self-storage apparatus(storage apparatus 15000B) corresponding to the volume information(corresponds to the logical volume 15001A). Then, the control I/Fprocessing unit 15003B calls the function processing unit 15004B toexecute prescribed function control to the logical volume 15001B.

It is thereby possible to execute prescribed function control to thelogical volume 15001A of the storage apparatus 15000A and the logicalvolume 15001B of the storage apparatus 15000B.

In this embodiment, although a case was explained relating to thereferral operation of the pair status of the local copy functionprovided by the storage apparatuses 15000A, 15000B, this embodiment canbe applied to the operation of various functions provided by the storageapparatuses 15000A, 15000B such as (1) other pair operations (paircreation, pair split, etc.) of the local copy function, (2) various pairoperations of the local copy function provided by the storageapparatuses 15000A, 15000B, (3) operation of the security function (LDEVguard function described later) to the logical volumes 15001A, 15001Bprovided by the storage apparatuses 15000A, 15000B, (4) operation of thelogical snapshot function (explained later) provided by the storageapparatuses 15000A, 15000B, and so on.

Incidentally, as a different mode of execution, upon receiving a commandto be issued to both storage apparatuses 15000A, 15000B on the activeside and standby side, a case may be considered where the active-sidestorage apparatus 15000A processes the received command and transfers itto the standby-side storage apparatus 15000B to perform the commandprocessing, and start both storage processing with a single command fromthe host 14000. Further, in the case of a command concerning theacquisition of the program status, a case may also be considered wherethe active-side storage apparatus 15000A that received the commandtransfers the same command to the standby-side storage apparatus 15000Bto acquire the status, and the active-side storage apparatus 15000Areturning the status to the command source after comparing bothstatuses.

(5) Fifth Embodiment

This embodiment describes a separate constitution of the function I/F.The constitution of this embodiment is explained with reference to FIG.15.

The constitution of this embodiment is roughly the same as theconstitution illustrated in FIG. 14. The differences with FIG. 14 are asfollowing three points:

(1) The command device 15002A and the command device 15002B are not apair of synchronous remote copy;

(2) The function management program 14003 recognizes the command device15002A and the command device 15002B as separate volumes 14004A, 14004B;and

(3) The function control request processing unit 14005 sends thefunction control request to the command device 15002A and the commanddevice 15002B.

In this embodiment, as with the fourth embodiment, the control requestreceived by the function control request processing unit 14005 containsinformation (also referred to as “apparatus information”) for uniquelyidentifying the control-target storage apparatuses 15000A, 15000B,information (also referred to as “volume information”) for uniquelyidentifying the control-target logical volumes 15001A, 150001B, andinformation incidental to the function control.

In this embodiment, unlike the fourth embodiment, as described above,the function control request processing unit 14005 that received thefunction control request from the user or another program in the host14000 or a program in a separate host from the host 14000 sends acontrol request to both command devices 15002A, 15002B.

Incidentally, the control request may also be rewritten such that thefunction control request processing unit 14005 determines the apparatusinformation, designates the logical volume 15001A as the volumeinformation to the command device 15002A, and designates the logicalvolume 15001B as the volume information to the command device 15002B.

Further, the user or another program in the host 14000 or a program in aseparate host from the host 14000 may identify the storage apparatuses15000A, 15000B, and issue different control requests in duplicate to thestorage apparatuses 15000A, 15000B. In other words, a control request ofthe logical volume 15001A is issued to the command device 15002A, and acontrol request of the logical volume 15001B is issued to the commanddevice 15002B.

(6) Sixth Embodiment

This embodiment describes a separate constitution of the function I/F.The constitution of this embodiment is explained with reference to FIG.16.

The sixth embodiment is roughly the same as the fourth embodiment. Thedifferences with the fourth embodiment are as follows.

(1) The host 14000, the storage apparatus 15000A, and the storageapparatus 15000B are mutually connected with an interconnection networksuch as a LAN (Local Area Network). Incidentally, these components maybe directly connected via a LAN, or connected via a switch.

(2) The constitution does not include a command device, and thecommunication among the three components (host 14000, storage apparatus15000A and storage apparatus 15000B) is conducted via the LAN.

(3) The function control request processing unit 14005 sends a controlrequest to the control I/F processing unit 15003A via the LAN.

(4) The control I/F processing unit 15003A that received the controlrequest sends a control request to the control I/F processing unit15003B via the LAN.

The point of processing the control request received by the control I/Fprocessing units 15003A, 15003B is the same as the fourth embodiment,and the sixth embodiment is able to provide an equivalent function I/Fas the fourth embodiment.

(7) Seventh Embodiment

This embodiment describes a separate constitution of the function I/F.The constitution of this embodiment is explained with reference to FIG.17.

The seventh embodiment is roughly the same as the sixth embodiment. Thedifferences with the sixth embodiment are as follows.

(1) The function control request processing unit 14005 sends a controlrequest to both control I/F processing units 15003A, 15003B via the LAN.

(2) The control I/F processing unit 15003A does not sends a controlrequest to the control I/F processing unit 15003B.

The point of processing the control request received by the control I/Fprocessing units 15003A, 15003B is the same as the sixth embodiment, andthe seventh embodiment is able to provide an equivalent function I/F asthe sixth embodiment.

(8) Eighth Embodiment

In this embodiment, a case is explained of applying a security function(LDEV security function) to the logical volumes in the storageapparatus.

FIG. 18 shows an embodiment of the LDEV security function. Theconstitution of this embodiment is roughly the same as FIG. 14. Thedifference with FIG. 14 is the addition of logical volume securityinformation 15005A, 15005B. The logical volume security information15005A, 15005B is used for access control from the host 14000 to thelogical volumes 15001A, 15001B in the storage apparatuses 15000A,15000B. As an example of access control, in order to inhibit thefalsification of data in the logical volumes 15001A, 15001B, there iscontrol for completely prohibiting the write access to the logicalvolumes 15001A, 15001B. Further, as a separate example, there is afunction of prohibiting writing for a prescribed period to dataobligated to be stored for a given period of time under laws andordinances. Moreover, as another example, there is a function ofprohibiting the read/write access from a specific host from theperspective of protecting confidential information.

Even in a constitution that seeks redundancy based on synchronous remotecopy using the two storage apparatuses 15000A, 15000B as shown in FIG.18, there are cases where it would be desirable to apply the LDEVsecurity function. In this case also, it is possible to control the LDEVsecurity function using the function I/F explained in the fourthembodiment. Specifically, it will suffice to set parameters concerningthe LDEV security in the logical volume security information 15005A,15005B storing security information of the target volume and referringto the same in the function processing unit 15004.

(9) Ninth Embodiment

In this embodiment, explained is a case of applying a logical copyfunction to the logical volumes in the storage apparatus.

A local copy function is the function of creating a replication of avolume designated by the user in the storage apparatus that is the sameas the copy source volume. The replication of the volume created usingthis function is accessed by the host for data mining or tape backup, orstored for a long time as backup data. When using the local copyfunction, a volume in which a replication is to be created and thevolume of the replication destination are designated as a pair (copypair), and a replication can be created by the user operating the pair.In the following explanation, the replication-target volume is sometimesreferred to as a primary volume, and the replication destination volumeis sometimes referred to as a secondary volume. In this embodiment, theavailability is improved by coordinating the local copy function withthe active-side storage apparatus and the standby-side storageapparatus.

FIG. 19 shows an embodiment of the local copy function. In FIG. 19, thehost 14000 is coupled to the storage apparatus 15000A and the storageapparatus 15000B. Further, the storage apparatus 15000A is coupled tothe external storage apparatus 16000A, and the storage apparatus 15000Bis coupled to the external storage apparatus 16000B. Moreover, the localcopy function and the differential bitmap (information showing thedifferential status between the primary volumes 15006A, 15006B and thesecondary volumes 15007A, 15007B) are executed and managed by thestorage apparatus 15000A and the storage apparatus 15000B.

This embodiment shows a constitution example where the primary volumes15006A, 15006B are in the storage apparatuses 15000A, 15000B, and thesecondary volumes 15007A, 15007B are in the external storage apparatuses16000A, 16000B. The primary volume 15006A and the secondary volume15007A are a pair, and the entity of the secondary volume 15007A is inthe external volume 16001A. Similarly, the primary volume 15006B and thesecondary volume 15007B are a pair, and the entity of the secondaryvolume 15007B is in the external volume 16001B.

<Operation in Duplex Status>

The Duplex status is one of the pair statuses, and is a status wherebackground copy described later is being performed from the primaryvolumes 15006A, 15006B to the secondary volumes 15007A, 15007B.

The read/write processing in the Duplex status is explained below.Incidentally, the following explanation of the read/write processing issubject to the active side of the target volumes (primary volumes15006A, 15006B) of the read/write processing being the storage apparatus15000A.

The read processing is foremost explained. The operating system 14001that received the read request from the application program 14002determines whether the active-side storage is the storage apparatus15000A or the storage apparatus 15000B based on the path managementfunction (in relation to the read-target primary volume), and issues aread request to the active-side storage apparatus 15000A. Theactive-side storage apparatus 15000A that received the read requestsends read data to the host 14000. The application program 14002receives the read-target data via the operating system 14001. This readprocessing is thereby complete.

Overview of the write processing is now explained. The operating system14001 that received the write request from the application program 14002determines whether the active-side storage apparatus is the storageapparatus 15000A or the storage apparatus 15000B based on the pathmanagement function (in relation to the write-target primary volume),and issues a write request to the active-side storage apparatus 15000A.The storage apparatus 15000A that received the write request receivesthe write data, stores the write data in the cache memory not shown, andsets the bit of the differential bit corresponding to the write data to1 (ON).

Further, the write data is thereafter copied (synchronous remote copy)from the cache memory in the storage apparatus 15000A to the primaryvolume 15006B in the storage apparatus 15000B based on the remote copyfunction. Incidentally, the method of synchronous remote copy is asexplained above. The storage apparatus 15000B that received the writedata from the storage apparatus 15000A based on synchronous remote copystores the write data in the cache memory not shown, and sets the bit ofthe differential bitmap corresponding to the write data to 1 (ON).Thereafter, the storage apparatus 15000B sends a write completion reportto the storage apparatus 15000A, and the storage apparatus 15000A thatreceived the write completion report sends a write completion report tothe host 14000.

Incidentally, the write data written respectively in the primary volume15006A of the storage apparatus 15000A and the primary volume 15006B ofthe storage apparatus 15000B is copied to the secondary volumes 15007A,15007B in asynchronously with the writing in the primary volumes 15006A,15006B (this processing is hereinafter referred to as “background copyprocessing”).

Background copy is performed by periodically monitoring the differentialbitmap, copying data of the area recorded as having a differential (inother words, the bit is ON) from the primary volumes 15006A, 15006B tothe secondary volumes 15007A, 15007B, and clearing the bit (OFF or 0)after the copy is complete.

Meanwhile, the standby-side storage apparatus 15000B also performssimilar processing triggered at the time the write data arrived based onsynchronous remote copy.

Incidentally, the present invention may adopt a constitution other thosedescribed above, and the primary volumes 15006A, 15006B may be in theexternal storage apparatus 16000A, or may be in the storage apparatuses15000A, 15000B. The same applies to the secondary volumes 15007A, 15007Bmay also be in the external storage apparatus 16000A, or in the storageapparatuses 15000A, 15000B.

When some kind of failure occurs and the read/write request to theprimary volume 15006A in the active-side storage apparatus 15000A can nolonger be processed, as described above, the operating system 14001continues access by switching the target of the read/write request tothe primary volume 15006B. In this case also, since a local copy pairexists in the storage apparatus 15000B, backup processing and the likedescribed above can be performed using the secondary volume 15007B.

<Operation of Pair Split and Suspend Status>

The Suspend status is one of the pair statuses, and indicates a statuswhere the image of the secondary volumes is decided. In this status,contents of the primary volumes and contents of the secondary volumes donot coincide, and the differential between the primary volumes and thesecondary volumes is managed with the differential bitmap. Further, inthis status, since the secondary volumes are in a stationary status, theuser is able to perform backup processing and the like described above.

The host 14000 stops the operation of background copy explained abovewhen making the pair of the Duplex status of local copy to a Suspendstatus (this is referred to as a “Pair Split”). The Pair Split isimplemented via the function I/F explained in the fourth to seventhembodiments.

(1) The host 14000 issues a stop command of local copy to the virtualstorage apparatuses 15000A, 15000B via the function I/F. Normally, onthe side of the host 14000, issuance of the I/O request is stoppedimmediately before the foregoing stop command.

(2) The active-side and standby-side virtual storage apparatuses 15000A,15000B complete the background copy of areas that are turned ON in thedifferential bitmap. The host 14000 receives a message indicating thatthe background copy in both virtual storage apparatuses 15000A, 15000Bis complete from the active-side storage apparatus 15000A or from bothstorage apparatuses 15000A, 15000B.

(3) The host 14000 receives the message, and thereafter resumes the I/Oissuance.

As a result of the processing up (2) above, the volume pair existingrespectively in the active-side and standby-side virtual storageapparatuses 15000A, 15000B becoming a Suspend status is confirmed. Atthis point, the pair status in both storage apparatuses 15000A, 15000Bwill be a Split status. Incidentally, the write location of the writerequest issued to the primary volume or the secondary volume during theSplit is recorded in the differential bitmap for the pairresynchronization described later.

The subsequent read/write request processing is roughly the same as theDuplex status. The difference from the Duplex status is that thebackground copy processing is not operated.

<Pair Creation>

The status where the primary volume and the secondary volume are not ofa pair relationship is referred to as a Simplex status. The processingfor changing the Simplex status to the Duplex status is referred to as apair creation. The transient state of changing the pair status from theSimplex status to the Duplex status is referred to as an Initial-Copyingstatus.

The pair creation command is implemented via the function I/F explainedwith reference to fourth to seventh embodiment.

(1) The host 14000 issues a pair creation command to the virtual storageapparatus 15000A via the function I/F. As a result, the pair creationprocessing is started in both the active-side and standby-side storageapparatuses 15000A, 15000B.

(2) Both storage apparatuses 15000A, 15000B set the pair status to anInitial-Copying status, turns ON all differential bitmaps, and startsbackground copy.

(3) When the background copy is completed until the end of thedifferential bitmap, the virtual storage apparatuses 15000A, 15000B setthe pair status to the Duplex status.

Incidentally, the read/write processing in the Initial-Copying status isthe same as the read/write processing in the Duplex status.

<Pair Resynchronization>

The operation of changing the pair status from a Suspend status to aDuplex status is referred to as pair resynchronization. The transientstatus of changing the pair status from the Suspend status to the Duplexstatus is referred to as a Duplex-Pending status.

The pair resynchronization command is implemented via the function I/Fexplained in the fourth to seventh embodiments.

(1) The host 14000 issues a pair resynchronization command to thestorage apparatus 15000A via the function I/F. As a result, the pairresynchronization processing is started at both the active-side andstandby-side storage apparatuses 15000A, 15000B.

(2) The both storage apparatuses 15000A, 15000B set the pair status toDuplex-Pending, and starts background copy.

(3) When the background copy is completed until the end of thedifferential bitmap, the storage apparatuses 15000A, 15000B set the pairstatus to the Duplex status.

The read/write processing in the Duplex-Pending status is the same asthe read/write processing in the Duplex status.

(10) Tenth Embodiment

This embodiment explains a local copy function that is different fromthe ninth embodiment. FIG. 20 shows a constitution example of theinformation system according to this embodiment.

Foremost, the difference in constitution between this embodiment and theninth embodiment is that there is no external storage apparatus 16000B,the storage apparatus 15000B and the external storage apparatus 16000Aare connected via an I/O network, and the entities of the secondaryvolumes 15007A, 15007B are all mapped to become the external volume16001A in the external storage apparatus 16000A. The remainingconstitution is the same as the ninth embodiment.

As a result of adopting the foregoing constitution, it is possible toeliminate the physical storage apparatus required by the secondaryvolumes 3000LB, 3000RB.

The major difference in the processing operation between this embodimentand the ninth embodiment is that the standby-side storage apparatus15000B does not perform background copy to the external volume 16001A,and only operates the pair status and differential bitmap 15010B ascontrol information concerning the pair through communication with thestorage apparatus 15000A.

The details of this processing operation are explained below.

<Operation in Duplex Status>

The read/write processing in the Duplex status is explained below.

The read processing is the same as the read processing of the ninthembodiment.

The write processing is as follows. The operating system 14001 thatreceived the write request from the application program 14002 determineswhether the active side is the storage apparatus 15000A or the storageapparatus 15000B based on the path management function (in relation tothe write-target primary volume), and issues a write request to theactive-side storage apparatus 15000A. The storage apparatus 15000A thatreceived the write request receives write data, stores the write data inthe cache memory, and sets the bit of the corresponding differentialbitmap 15010A to 1 (ON).

Thereafter, the write data is copied from the primary volume 15006A inthe storage apparatus 15000A to the primary volume 15006B in the storageapparatus 15000B based on the synchronous remote copy function.Incidentally, the method of synchronous remote copy is as describedabove. The storage apparatus 15000B that received the write data fromthe storage apparatus 15000B based on the synchronous remote copyfunction stores the write data in the cache memory, and sets the bit ofthe differential bitmap 15010B corresponding to the write data to 1(ON). The storage apparatus 15000B thereafter sends a write completionreport to the storage apparatus 15000A, and the storage apparatus 15000Athat received the write completion report sends a write completionreport to the host 14000.

Incidentally, the data written into the primary volume 15006A of thestorage apparatus 15000A is background-copied to the secondary volume15007A asynchronously with the writing into the primary volume 15006A.Unlike the write processing in the ninth embodiment, the data writteninto the primary volume 15006B of the storage apparatus 15000B is notsubject to background copy.

The background copy processing in the storage apparatus 15000Aperiodically monitors the differential bitmap 15010A, copies the data ofareas recorded as having a differential (in other words, bit is ON) fromthe primary volume 15006A to the secondary volume 15007A, and clears thebit after the copy is complete (OFF or 0). Incidentally, thisembodiment, unlike the write processing in the ninth embodiment,background copy is not performed on the side of the storage apparatus15000B.

Subsequently, unlike the write processing in the ninth embodiment, thestorage apparatus 15000A notifies the location information of thecleared bit to the storage apparatus 15000B. The storage apparatus15000B that received the notice clears the bit (differential bit) in thestorage apparatus 15000B corresponding to the foregoing bit.Incidentally, notification of the location information of thedifferential bit is conducted via a command device in the storageapparatus 15000B. Nevertheless, in a constitution where the storageapparatuses 1500A, 15000B are connected via a LAN, the notification mayalso be made through communication via the LAN. In the followingexplanation, let it be assumed that the communication concerning thecontrol information of functions such as the differential bit and pairstatus between the storage apparatus 15000A and the storage apparatus15000B is conducted via the command device or the LAN.

When some kind of failure occurs and the I/O request to the active-sideprimary volume 15006A can no longer be processed, the operating system14001, as with the ninth embodiment, continues access by switching thetarget of the I/O request to the primary volume 15006B.

<Operation of Pair Split and Suspend Status>

When the host 14000 is to change the pair of the Duplex status of localcopy to a Split status, it performs pair split as in the ninthembodiment. Incidentally, although stop processing of background copy isperformed in pair split, in this embodiment, stop processing is notactually performed since background copy is not performed in the storageapparatus 15000B.

The subsequent I/O request processing is roughly the same as in theDuplex status. The difference with the Duplex status is that thebackground copy processing does not operate in the storage apparatus15000B.

<Pair Creation>

The pair creation command is the same as the ninth embodiment in that itis implemented via the function I/F explained in the fourth to seventhembodiments.

(1) The host 14000 issues a pair creation command to the storageapparatus 15000A via the function I/F. As a result, the pair creationprocessing is started at both the active-side and standby-side storageapparatuses 15000A, 15000B.

(2) Both storage apparatuses 15000A, 15000B set the pair status to anInitial-Copying status. The storage apparatus 15000A turns ON alldifferential bitmaps 15010A and starts background copy. Unlike the ninthembodiment, the storage apparatus 15000B turns ON all differentialbitmaps 15010B, but does not perform background copy.

(3) Operation for clearing the differential bit corresponding to areasto which background copy in the storage apparatus 15000A is complete,and the incidental operations (notification of the location informationof the differential bit and clearing of the differential bit) are thesame as the operations in the Duplex status.

(4) Unlike the ninth embodiment, when the background copy is completeduntil the end of the differential bitmap 15010A, the storage apparatus15000A sets the pair status to a Duplex status, and notifies the storageapparatus 15000B that the pair status has changed to a Duplex status.The storage apparatus 15000B that received the notification sets thepair status to a Duplex status.

The read/write processing in the Initial-Copying status is the same asthe read/write processing in the Duplex status.

<Pair Resynchronization>

The pair resynchronization command is the same as the ninth embodimentin that it is implemented via the function I/F described in the fourthto seventh embodiments.

(1) The host 14000 issues a pair resynchronization command to thestorage apparatus 15000A via the function I/F. As a result, the pairresynchronization processing is started at both the active-side andstandby-side storage apparatuses 15000A, 15000B.

(2) The storage apparatus 15000A sets the pair status to aDuplex-Pending status, and starts background copy. Unlike the ninthembodiment, the storage apparatus 15000B does not perform backgroundcopy.

(3) When the background copy is completed until the end of thedifferential bitmap 15010A, the storage apparatus 15000A sets the pairstatus to a Duplex status. Nevertheless, unlike the ninth embodiment,only the storage apparatus 15000A performs this processing. The storageapparatus 15000A thereafter notifies the storage apparatus 15000B thatthe pair status has changed to a Duplex status. The storage apparatus15000B that received the notification sets the pair status to a Duplexstatus.

The read/write processing in the Duplex-Pending status is the same asthe read/write processing in the Duplex status.

(11) Eleventh Embodiment

Constitution of an AOU (Allocation On Use) function is now explained.The AOU function is a function of allocating a real storage area onlyregarding the area used by (written from) the host.

The AOU function is constituted of a pool as an aggregate of realvolumes that actually store data, and a virtual volume as a volume to bepresented to the host. The virtual volume in this embodiment is virtualfrom the perspective that real data in which writing was performed isonly allocated. Real data is not allocated to the entire address spaceof the volume presented to the host. Incidentally, the real volume maybe in the external storage apparatus, or may be in the same storageapparatus as the virtual volume.

FIG. 21 shows an embodiment of the AOU function. In FIG. 21, the host14000 is coupled to the storage apparatus 15000A and the storageapparatus 15000B. Further, the storage apparatus 15000A is coupled tothe external storage apparatus 16000A, and the storage apparatus 15000Bis coupled to the external storage apparatus 16000B.

This embodiment shows a constitution example where the real volume16002A is in the external storage apparatuses 16000A, 16000B. Data inthe virtual volume 15008A is associated with data in the real volume16002A of the pool 16003A. Similarly, data in the virtual volume 15008Bis associated with data in the real volume 16002B of the pool 16003B.Further, the virtual volume 15008A and the virtual volume 15008B areconstituted so that their contents coincide based on the synchronousremote copy function. The method of synchronous remote copy is asdescribed above.

The read/write processing of this constitution is now described.Incidentally, the following explanation on the read/write processing isbased on the premise that the active side of the target volume of theread/write processing is the storage apparatus 15000A.

Read processing is foremost explained. The operating system 14001 thatreceived the read request from the application program 14002 determineswhether the active-side storage is the storage apparatus 15000A or thestorage apparatus 15000B based on the path management function, andissues a read request to the active-side storage apparatus 15000A. Thestorage apparatus 15000A that received the read request refers to thevirtual address/real address mapping table 15009A, and determineswhether a real area in the pool 16003A is allocated to the read data.

When real data is allocated in the foregoing determination, the storageapparatus 15000A reads the read data from the real area and sends it tothe host 14000. The application 14002 receives the read data via theoperating system 14001. The read processing is thereby complete.

Write processing is now explained. The operating system 14001 thatreceived a write request from the application program 14002 decideswhether the active-side storage apparatus is the storage apparatus15000A or the storage apparatus 15000B based on the path managementfunction, and issues a write request to the active-side storageapparatus 15000A. The storage apparatus 15000A that received the writerequest refers to the virtual address/real address mapping table 15009A,and determines whether a real area in the pool 16003A is allocated tothe write-target data (determination 200).

If a real area is allocated in the foregoing determination, the storageapparatus 15000A receives write data from the host 14000, and stores thewrite data in the area in the cache memory not shown corresponding tothe real area. The storage apparatus 15000A thereafter sends a writerequest for write data to the storage apparatus 15000B based on thesynchronous remote copy function. The storage apparatus 15000B thatreceived the write request from the storage apparatus 15000A determineswhether a real area in the pool 16003A is allocated to the write data.Here, since contents of the virtual volume 15008A and contents of thevirtual volume 15008B coincide based on the synchronous remote copyfunction, it is determined that the real area is allocated.Subsequently, the storage apparatus 15000B receives write data from thestorage apparatus 15000A, and stores the write data in the area in thecache memory not shown corresponding to the real area, and issues awrite completion report to the storage apparatus 15000A.

If a real area is not allocated in the foregoing determination(determination 200), the storage apparatus 15000A registers the addressof write data in the virtual address/real address mapping table 15009A,and thereby allocates a real area. Subsequently, the storage apparatus15000A receives write data from the host 14000, and stores the writedata in the area in the cache memory not shown corresponding to the realarea. The storage apparatus 15000A thereafter sends a write request forwrite data to the storage apparatus 15000B based on the synchronousremote copy function. The storage apparatus 15000B that received thewrite request from the storage apparatus 15000A determines whether areal area in the pool 16003B is allocated to the write data. Here, sincecontents of the virtual volume 15008A and contents of the virtual volume15008B coincide based on the synchronous remote copy function, it isdetermined that the real area is allocated.

Subsequently, the storage apparatus 15000B receives write data from thestorage apparatus 15000A, and stores the write data in the area in thecache memory not shown corresponding to the real area, and issues awrite completion report to the storage apparatus 15000A. Subsequently,the storage apparatus 15000B registers the address of write data in thevirtual address/real address mapping table 15009B, and thereby allocatesa real area. The storage apparatus 15000B receives write data from thestorage apparatus 15000A, stores the write data in the area in the cachememory not shown corresponding to the real area, and thereafter issues awrite completion report to the storage apparatus 15000A. The storageapparatus 15000A that received the write completion report issues awrite completion report to the host 14000. The host 14000 receives thewrite completion report, and the write processing is thereby complete.

Incidentally, data stored in the cache memory is written into the realvolumes 16002A, 16002B asynchronously with the storage of such data inthe cache memory.

When some kind of failure occurs and the application 14002 is no longerable to perform the read/write processing via virtual volume 15008A inthe storage apparatus 15000A, the path management function provided bythe operating system 14001 detects a failure, and switches the accesspath of the read/write processing to go through the virtual volume15008B in the storage apparatus 15000B. Since contents of the virtualvolume 15008A and contents of the virtual volume 15008B coincide basedon the synchronous remote function, even when the access path isswitched, read/write processing can be ongoingly performed in a normalmanner.

(12) Twelfth Embodiment

This embodiment describes an embodiment that is different from the AOLfunction of the eleventh embodiment. FIG. 22 shows a constitutionexample of this embodiment.

The difference in the constitution of this embodiment and the eleventhembodiment is that there is no external storage apparatus 16000B, andthe real areas of the virtual volumes 15008A, 15008B are all allocatedto areas in the pool 16003A in the external storage apparatus 16000A.The remaining constitution is the same as the eleventh embodiment.

Incidentally, in this embodiment, since the storage apparatus 15000A andthe storage apparatus 15000B use the real volume 16002A in the commonexternal storage apparatus 16000A as the common pool, unlike theeleventh embodiment, this embodiment is limited to a constitution wherethe real volume 16002A is in the external storage apparatus 16000A.

As a result of adopting the foregoing constitution, it is possible toeliminate the physical storage apparatus (HDD or the like) required bythe pool.

The major difference in the processing operation between this embodimentand the eleventh embodiment is in that the standby-side storageapparatus 15000B does not perform writing from the cache memory into thereal volume 16002A of the external storage apparatus 16000A, and thatthe storage apparatus 15000A notifies the storage apparatus 15000B ofthe update to the virtual address/real address mapping table 15009A, andthe storage apparatus 15000B that received the notification updates thevirtual address/real address mapping table 15009B.

Details of the processing operation are explained below.

Foremost, the read processing is the same as the read processing in theeleventh embodiment.

Write processing is now explained. The operating system 14001 thatreceived a write request from the application program 14002 decideswhether the active-side storage apparatus is the storage apparatus15000A or the storage apparatus 15000B based on the path managementfunction, and issues a write request to the active-side storageapparatus 15000A. The storage apparatus 15000A that received the writerequest refers to the virtual address/real address mapping table 15009A,and determines whether a real area in the pool 16003A is allocated tothe write-target data (determination 300).

If a real area is allocated in the foregoing determination, the storageapparatus 15000A receives write data from the host 14000, and stores thewrite data in the area in the cache memory corresponding to the realarea. The storage apparatus 15000A thereafter sends a write request forwrite data to the storage apparatus 15000B based on the synchronousremote copy function. In this embodiment, unlike the eleventhembodiment, the storage apparatus 15000B that received the write requestfrom the storage apparatus 15000A instantaneously receives write datafrom the storage apparatus 15000A, stores such data in the cache memory,and thereafter issues a write completion report to the storage apparatus15000A. The storage apparatus 15000A that received the write completionreport from the storage apparatus 15000B sends a write completion reportto the host 14000.

If a real area is not allocated in the foregoing determination(determination 300), the storage apparatus 15000A registers the addressof write data in the virtual address/real address mapping table 15009A,and thereby allocates a real area. Subsequently, the storage apparatus15000A receives write data from the host 14000, and stores the writedata in the area in the cache memory not shown corresponding to the realarea. The storage apparatus 15000A thereafter sends a write request forwrite data to the storage apparatus 15000B based on the synchronousremote copy function.

In this embodiment, unlike the eleventh embodiment, the storageapparatus 15000B that received the write request from the storageapparatus 15000A instantaneously receives write-target data from thestorage apparatus 15000A, stores such data in the cache memory, andthereafter issues a write completion report to the storage apparatus15000A. The storage apparatus 15000A receives the write completionreport from the storage apparatus 15000B, and thereafter sends thecontents of change in the virtual address/real address mapping table15009A to the storage apparatus 15000B.

The storage apparatus 15000B that received the contents of change in thevirtual address/real address mapping table 15009A makes similar changesto the virtual address/real address mapping table 15009B. Thereby, thereal area of the write area in the virtual volume 15008B of the storageapparatus 15000B will be mapped to the real area (allocated by thestorage apparatus 15000A) in the real volume 16002A of the commonexternal storage apparatus 16000A. The storage apparatus 15000B notifiesthe storage apparatus 15000A of the update of the virtual address/realaddress mapping table 15009B. Subsequently, the storage apparatus 15000Athat received the notification issues a write completion report to thehost 14000. Incidentally, the storage apparatus 15000A simultaneouslymay perform (1) the data transmission of synchronous remote copy, and(2) the transmission of change of contents in the virtual address/realaddress mapping table 15009A, receive the completion report on theprocessing of (1) and (2), and thereafter issue a write completionreport to the host 14000. Subsequently, the host 14000 receives thewrite completion report, and the write processing thereby complete.

Incidentally, data stored in the cache memory of the storage apparatus15000A is written (destaged) into the real volume 16002A with thestorage apparatus 15000A asynchronously with the storage in the cachememory. After destaging is complete, the storage apparatus 15000Anotifies such completion to the storage apparatus 15000B. The storageapparatus 15000B that received the notification discards the area of thecache memory corresponding to the writing. Incidentally, instead ofdiscarding the area, the area of the cache memory corresponding to thewriting may be cleaned (a status where contents of the cache memorycoincide with data in the storage apparatus (such as an HDD)).

When some kind of failure occurs and the application 14002 is no longerable to perform the read/write processing via virtual volume 15008A inthe storage apparatus 15000A, the path management function provided bythe operating system 14001 detects a failure, and switches the accesspath of the read/write processing to go through the virtual volume15008B in the storage apparatus 15000B. Since contents of the virtualvolume 15008A and contents of the virtual volume 15008B coincide basedon the synchronous remote function, even when the access path isswitched, read/write processing can be ongoingly performed in a normalmanner.

(13) Thirteenth Embodiment

In this embodiment, an example is explained where the logical snapshotfunction is applied to the volumes in the storage apparatuses.

A logical snapshot function is a function that is similar to the localreplication function, and a function for providing the user withreplicated data at the point-in-time designated by the user.Nevertheless, the secondary volume having replicated data is a virtualvolume provided using the write data subsequent to the replicationcreation command stored in the area of the real volume belonging to thepool, and data of the primary volume. The entity of the virtualsecondary volume is retained in a pool that is an aggregate of realvolumes. The relationship of the primary volume and the secondary volumemay be referred to as a snapshot pair or simply as a pair. In thelogical snapshot function, from the perspective that a logical volumehaving the same contents as the primary volume at the stationary pointis not actually created, the secondary volume is virtual. The logicalsnapshot function, unlike the local copy function described above, doesnot need a secondary volume that is the same size as the size of theprimary volume. Thereby, it is possible to eliminate storage apparatuses(HDDs and the like) required for retaining the contents of the secondaryvolume.

In this embodiment, the availability can also be improved bycoordinating the active-side storage apparatus and the standby-sidestorage apparatus regarding this logical snapshot function.

FIG. 23 shows an embodiment of the snapshot function. In FIG. 23, thehost 14000 is coupled to the storage apparatus 15000A and the storageapparatus 15000B. Further, the storage apparatus 15000A is coupled tothe external storage apparatus 16000A, and the storage apparatus 15000Bis coupled to the external storage apparatus 16000B. Further, thesnapshot function and the differential bitmaps 15010A, 15010B(information showing the status of differential between the primaryvolumes 15006A, 15006B at the stationary point and the primary volumes15006A, 15006B at the current time) and the virtual address/real addressmapping tables 15009A, 15009B (tables for managing the location of theentity of the virtual secondary volumes 15007A, 15007B) are executed andmanaged by the storage apparatus 15000A and the storage apparatus15000B. Further, the primary volume 15006A in the storage apparatus15000A and the primary volume 15006B in the storage apparatus 15000B areconfigured to form a remote copy pair.

This embodiment shows a constitution example where the primary volumes15006A, 15006B are in the storage apparatuses 15000A, 15000B and thepool 16003A, 16003B are in the external storage apparatus 16000A,16000B. Incidentally, the pools 16003A, 16003B may also be in thestorage apparatuses 15000A, 15000B.

<Logical Snapshot Creation Command>

When the user using the host 14000 issues a logical snapshot creationcommand, a creation command is issued to the active-side storageapparatus 15000A and the standby-side storage apparatus 15000B accordingto the methods described in the previous embodiments. The storageapparatuses 15000A, 15000B that received the creation command preparevirtual secondary volumes 15007A, 15007B, and allocate the differentialbitmaps 15010A, 15010B that are all 0 (meaning no differential) and thevirtual address/real address mapping tables 15009A, 15009B to thesecondary volumes 15007A, 15007B.

<Read Processing to Primary Volume>

This is the same as the previous embodiments.

<Write Processing to Primary Volume>

The operating system 14001 that received the write request from theapplication program 14002 determines whether the active-side storage isthe storage apparatus 15000A or the storage apparatus 15000B based onthe path management function (in relation to the write-target primaryvolume), and issues a write request to the active-side storage apparatus15000A. The storage apparatus 15000A that received the write requestchecks the differential bitmap 15010A of the write-target address. Ifthe result is 1, data sent from the host 14000, together with the writerequest, is stored as write data of the primary volume in the cachememory. Meanwhile, if the result is 0, the following Copy-On-Writeprocessing is performed for using the before-updated data of the primaryvolume 15006A as data for the secondary volume 15007A.

(Step 1) The storage area of the real volume 16002A belonging to thepool 16003A is allocated.

(Step 2) The before-updated data is copied from the primary volume15006A to said storage area while using the cache memory.

(Step 3) The pool management information for managing the storagedestination of the before-updated data to be saved is updated toindicate which area of the real volume 16002A in the pool 16003A thatdata has been stored.

(Step 4) The received write data is stored as data to the address of theprimary volume 15006A in the cache memory, and a write completion replyis returned.

In parallel with this, the write data is copied from the primary volume15006A in the storage apparatus 15000A to the primary volume 15006B inthe storage apparatus 15000A based on the remote copy function, andsimilar processing is performed. Thus, the storage apparatuses 15000A,15000B respectively manage the virtual address/real address mappingtables 15009A, 15009B and the differential bitmap s15010A, 15010B.

<Read Processing to Secondary Volume>

The operating system 14001 that received the write request from theapplication program 14002 determines whether the active side is thestorage apparatus 15000A or the storage apparatus 15000B based on thepath management function (in relation to the read-target secondaryvolume), and issues a read request to the active-side storage apparatus15000A. The storage apparatus 15000A that received the read requestchecks the differential bitmap 15010A recorded in the primary volume15006A. As a result, if the bit of the read-target address is 0, datastored in the same address of the primary volume 15006A is returned tothe host 14000, and the operating system 14001 returns the data to theapplication 14002. Meanwhile, if the bit of the read-target address is1, the operating system 14001 refers to the virtual address/real addressmapping table 15009A, decides the location of the pre-updated dataconcerning the read-target address of the primary volume 15006A, andreturns the data from the real volume belonging to the pool 16003A tothe host 14000 (application program 14002).

<Write Processing to Secondary Volume>

In the host 14000, the operating system 14001 that received the writerequest from the application program 14002 determines whether the activeside is the storage apparatus 15000A or the storage apparatus 15000Bbased on the path management function (in relation to the write-targetsecondary volume), and issues a write request to the active-side storageapparatus 15000A. The storage apparatus 15000A that received the writerequest checks the differential bitmap 15010A of the write-targetaddress allocated to the primary volume 15006A. If the result is 1, byreferring to the virtual address/real address mapping table 15009A, theoperating system 14001 searches the storage area of a real volume 16002Ain the pool 16003A storing the before-updated data of the address of theprimary volume 15006A, and stores the write data in the area. Meanwhile,when the result is 0, the following processing is performed.

(A) An area of the real volume 16002A belonging to the pool 16003A isallocated.

(B) Write data is stored in the allocated area and the virtualaddress/real address mapping table 15009A is updated in order toindicate which area of the real volume 16002A in the pool 16003A thatdata has been stored.

(C) The bit corresponding to the address of the differential bitmap15010A is updated to 1.

In parallel with this, the write data is copied from the primary volume15006A in the storage apparatus 15000A to the primary volume 15006B inthe storage apparatus 1000R based on the remote copy function, andsimilar processing is performed. Thus, the storage apparatuses 15000A,15000B respectively manage the virtual address/real address mappingtables 15009A, 15009B and the differential bitmaps 15010A, 15010B.

<Copy-After-Write Processing>

The storage apparatuses 15000A, 15000B may execute the followingCopy-After-Write processing in substitute for the Copy-On-Writeprocessing to be executed upon writing into the primary volumes 15006A,15006B.

(Step 1) The received write data is stored as data to the address of theprimary volumes 15006A, 15006B in the cache memory, and a writecompletion reply is returned. However, destaging of the write data isinhibited.

(Step 2) Storage areas of the real volumes 16002A, 16002B belonging tothe pools 16003A, 16003B are allocated.

(Step 3) The before-updated data is coped from the primary volumes15006A, 15006B to said storage area while using the cache memory.

(Step 4) The pool management information for managing the storagedestination of the saved before-updated data is updated to indicatewhich area of the real volumes 16002A, 16002B in the pools 16003A,16003B that data has been stored.

(Step 5) Destaging of write data that was inhibited is permitted.

<Failure>

When some kind of failure occurs and the read/write request to theactive-side primary volume 15006A and the secondary volume 15007A can nolonger be processed, as described above, the operating system 14001 isable to continue access by switching the read/write request target tothe standby-side primary volume 15006B or the secondary volume 15007B.Incidentally, as described above, in order to issue a write request tothe same storage apparatuses 15000A, 15000B, preferably, the primaryvolumes 15006A, 15006B and the secondary volumes 15007A, 15007B of thesnapshot function simultaneously switch the secondary volumes 15007A,15007B when switching of the primary volumes 15006A, 15006B is required,and contrarily switch the primary volumes 15006A, 15006B as well whenswitching of the secondary volumes 15007A, 15007B is required.

(14) Fourteenth Embodiment

An embodiment of a logical snapshot function that is different from thethirteenth embodiment is now explained. FIG. 24 shows one constitutionexample of this embodiment.

Foremost, the difference in the constitution between this embodiment andthe thirteenth embodiment is that the external storage apparatus 16000Bdoes not exist, and the real area of the virtual secondary volumes15007A, 15007B is allocated to the area in the pool 16003A of anyexternal storage apparatus 16000A. The remaining constitution is thesame as the thirteenth embodiment.

Incidentally, in this embodiment, since the storage apparatus 15000A andthe storage apparatus 15000B use the real volume 16002A in the commonexternal storage apparatus 16000A as the common pool 16003A, and, unlikethe thirteenth embodiment, the real volume 16002A is limited to theconstitution within the external storage apparatus 16000A.

As a result of adopting the foregoing constitution, it is possible toeliminate the physical storage apparatus (such as an HDD) required bythe pool 16003A.

The major difference of the processing operation in this embodiment andthe thirteenth embodiment is as follows.

(A) In normal situations, in substitute for the standby-side storageapparatus 15000B not performing writing from the cache memory into thereal volume 16002A of the external storage apparatus 16000A, the timingthat the active-side storage apparatus 15000A destages the datacorresponding to the real volume 16002A in the primary volume 15006A,the secondary volume 15007A, and the pool 16003A is conveyed to thestandby-side storage apparatus 15000B, and the standby-side storageapparatus 15000B thereby discards the data in the cache memory.

(B) The storage apparatus 15000A notifies the storage apparatus 15000Bof the update to the virtual address/real address mapping table 15009A,and the storage apparatus 15000B that received the notification updatesthe virtual address/real address mapping table 15009B.

Further, in substitute of the processing of (A), caching of datacorresponding to the real volume 16002A in the secondary volumes 15007A,15007B or the pool 16003A can be invalidated. Here, since the saving ofthe before-updated data by the foregoing Copy-On-Write processingincludes the storing of data in the real volume 16002A in the pool16003A until the writing in the primary volumes 15006A, 15006B iscomplete, the performance will deteriorate. But since this does notoccur in the Copy-After-Write mode, this is preferable.

Several embodiments of the present invention were described above, butthese embodiments are merely illustrations for explain the presentinvention and are not intended to limit the scope of invention in anyway. The present invention may be worked in various other modes withoutdeviating from the gist of this invention. For example, a nonvolatilememory can be used in substitute for the HDD 1030 and the cache memory1020. As the nonvolatile memory, for example, various types ofnonvolatile memories such as a flash memory (specifically, for instance,a NAND-type flash memory), MRAM (Magnetoresistive Random Access Memory),and PRAM (Parameter Random Access Memory) can be used.

(15) Fifteenth Embodiment

FIG. 28, in which the same reference numeral is given to the componentscorresponding to those illustrated in FIG. 1, is a schematic diagramshowing an embodiment of applying network switches (virtualizationswitches) 28000L, 28000R having a virtualization function.

<1. Hardware Constitution of Present Embodiment>

The virtualization switches 28000L, 28000R have a plurality of networkports, and processors for controlling the network ports control thetransfer of the respective ports, detect failures, and performvirtualization described later. Incidentally, although not shown in thisschematic diagram, as with the first embodiment explained with referenceto FIG. 1, a management host is coupled to the virtualization switches28000L, 28000R, and a user is able to configure settings in thevirtualization switches 28000L, 28000R or perform setting copy betweenthe virtualization switches 28000L, 28000R via this management host.Incidentally, the other components are the same as the first tofourteenth embodiments, and the explanation thereof is omitted.

<2. Characteristics of Present Embodiment Employing VirtualizationSwitch>

Virtualization provided by the virtualization switches 28000L, 28000Rdiffer from the first to fourteenth embodiments by comprising thefollowing characteristics.

(Characteristic 1) A virtual WWN (or port name) can be provided. Theport of the fibre channel switch is referred to as an F port or an Eport, and has an attribute that is different from the N port (meaningthe start point or end point of communication) of the normal host or thestorage. Thus, when performing virtualization in the virtualizationswitches 28000L, 28000R, if a virtual WWN, which is not actuallyconnected internally, is created and provided to both virtualizationswitches 28000L, 28000R, software in the host 1100 will no longer haveto explicitly switch the I/O path. Incidentally, more specifically,although communication of the fibre channel is conducted with theforegoing port name, this is an identifier allocated by the fibrechannel switch, and the identifier internally contains information foridentifying the switch to be used in the routine. Thus, bothvirtualization switches 28000L, 28000R perform the routine by allocatinga port name so to simulate to the host 1100 as the N port having avirtual WWN is coupled to the both virtualization switches 28000L,28000R via the virtual switch.

(Characteristic 2) Caching is not performed in the switches. The fibrechannel switch normally decides the transfer destination by referring toonly the control header and performs transfer control in a so-calledcut-through method where data buffering is not performed, caching isoften not performed even when providing a virtualization function.Incidentally, when performing caching, processing related to thischaracteristic is realized with the processing as with the embodimentsdescribed above. Further, the read/write processing of thevirtualization switches 28000L, 28000R in a case of not performingcaching can be considered to be similar to write-through-type control oftransferring a request to the storage apparatus 1500L upon receiving anI/O request and returning a processing completion report to the host1100 upon waiting for the processing to be complete.

(Characteristic 3) High availability in this embodiment is realizedmerely by setting the same virtualization in both virtualizationswitches 28000L, 28000R. This is possible because caching is notperformed in the virtualization switches 28000L, 28000R. Incidentally,when the virtualization switches 28000L, 28000R are to perform remotecopy or local copy, and there is information such as differentialbitmaps in the switches, as with the previous embodiments, it isnecessary to retain internal information in both the primary andsecondary systems.

Incidentally, although the virtualization switches 28000L, 28000R wereexplained above as being fibre channel switches, the virtualizationswitches 28000L, 28000R may also be switches employing Ethernet(registered trademark), or iSCSI or TCP/IP. Here, the WWN may correspondto the MAC address and the port name may correspond to the IP address,but in cases of Ethernet (registered trademark) or TCP/IP, the routinemay be performed to the IP address by directly providing the virtualport and the IP address allocated thereto externally without providingvirtual switches.

(16) Sixteenth Embodiment

The sixteenth embodiment is now explained. This embodiment relates to aninvention of providing a virtual storage apparatus of configuring theAOU function explained in the eleventh embodiment and twelfth embodimentin a high availability constitution. Incidentally, functions and thelike that are not explained below are the same as theconfiguration/constitution of the information system explained in thefirst to fifteenth embodiments.

As described above, regarding the volume (hereinafter referred to as the“AOU volume”) to be provided by the virtual storage apparatus to thehost 1100 based on the foregoing function, the AOU function is afunction of allocating a storage area of a HDD (more specifically, apart or the whole of a storage area of a volume (hereinafter referred toas a “pool volume”) constituted to a HDD) to an address written into bythe host 1100, instead of allocating the storage area of the HDD to alladdress of the AOU volume from the start of use. The AOU function isable to effectively use the HDD. This is because, it is not able todynamically expand the data capacity during ongoing access with acertain type of file system operating in the host 1100, theadministrator of the host 1100 sets the volume capacity by including thedata capacity that may be used in the future. Thus, conventionaltechnology had to be equipped with a HDD that will not be used at thetime the volume capacity is set, even though it may not be used in thefuture.

Incidentally, from the perspective of effectively using the HDDcapacity, it is preferable that the area of the pool volume isunallocated to the area of the AOU volume unwritten, but this embodimentis not limited thereto if there is another objective (realizing highperformance, etc.).

<1. Overview of Present Embodiment>

FIG. 29, in which the same reference numeral is given to the componentscorresponding to those illustrated in FIG. 1, shows an overview of thisembodiment. With the information system according to this embodiment,two virtual storage apparatuses 1000L, 1000R are coupled to the commonstorage apparatus 1500L. As a result of the two virtual storageapparatuses 1000L, 1000R of high availability having the AOU function,the service down time of the information system is shortened.Incidentally, unless specified herein, the storage apparatus 1500L is inan accessible state from both virtual storage apparatuses 1000L, 1000R;in other words, it is in a shared state, and with the existence of astorage apparatus that is not shared, the volume in the storageapparatus can be used as the storage area of AOU. Further, although notshown in FIG. 29, in the case of this embodiment, as with the firstembodiment, a management host 1200 (FIG. 1) is coupled to the virtualstorage apparatuses 1000L, 1000R.

Sections that are different from the foregoing embodiments are mainlyexplained below. The two virtual storage apparatuses 1000L, 1000R usethe AOU address mapping information 31030L, 31030R to create the AOUvolumes 29010L, 29010R, and provide these to the host 1100. The AOUaddress mapping information 31030L, 31030R contains the correspondenceof the address space of the AOU volumes 29010L, 29010R and the area ofthe pool volume in the virtual storage apparatuses 1000L, 1000R or thearea of the pool volume in the storage apparatus 1500L.

When a write request is issued from the host 1100 to the AOU volume29010L, the primary virtual storage apparatus 1000L determines whetheran area of the pool volume is allocated to the address range of therequest target, and, if not allocated, it allocates the area of the poolvolume of the virtual storage apparatus 1000L or the storage apparatus1500L. Then, as a result of the write request being processed, writedata is stored in the cache area of the primary virtual storageapparatus 1000L. Further, write data to the AOU volume 29010L istransferred to the secondary virtual storage apparatus 1000R based onthe synchronous remote copy, and write data is stored in the cache areaas with the primary system.

Subsequently, both virtual storage apparatuses 1000L, 1000R performdestaging processing, but only one of the virtual storage apparatuses1000L, 1000R performs destaging to the write data corresponding to thestorage apparatus 1500L. This is because if both virtual storageapparatuses 1000L, 1000R independently destage the write data, datastored in the storage apparatus 1500L will become inconsistent (forinstance, data loss or inconsistency of write sequence such as the lastwritten data being deleted and returning to the previous write data).Thus, it is necessary to decide in advance which one of the virtualstorage apparatuses 1000L, 1000R will perform destaging before destagingbecomes required. This embodiment explains an example of this where theprimary virtual storage apparatus 1000L performs destaging, butdestaging may also be performed by the secondary virtual storageapparatus 1000R, or the virtual storage apparatus 1000L, 1000R toperform such destaging may be decided based on the address space of thedestaging target.

In the case of a read request also, the primary virtual storageapparatus 1000L foremost determines whether an area of the pool volumeis allocated to the address range of the request target. When an area isallocated as a result of this determination, the virtual storageapparatus 1000L reads data from the area (including data in the cachememory not shown) of the corresponding pool volume and transfers this tothe host 1100. When an area is not allocated, the virtual storageapparatus 1000L returns a predetermined value (zero, for instance).

FIG. 30 is a schematic diagram after switching the I/O requestprocessing to the secondary virtual storage apparatus 1000R subsequentto the function of causing the primary virtual storage apparatus 1000Lto stop. As illustrated in FIG. 30, the secondary virtual storageapparatus 1000R processes the I/O request based on the AOU addressmapping information 31030R in the storage apparatus 1500L and thevirtual storage apparatus 1000L. Thus, as a result of the primary andsecondary virtual storage apparatuses 1000L, 1000R communicating on aregular basis, they maintain the contents of the AOU address mappinginformation 31030L, 31030R that are the same as the portions relating tothe storage apparatus 1500L. Thereby, the secondary virtual storageapparatus 1000R is able to take over the allocation status of thestorage apparatus 1500L. Further, the secondary virtual storageapparatus 1000R does not delete data stored in the cache memory fromsuch cache memory unless it is data that has been destaged from thecache memory in the primary virtual storage apparatus 1000L. Thereby, itis possible to prevent data loss even if the data from the cache memoryin the primary virtual storage apparatus 1000L is volatilized when thefunction is stopped.

<2. Programs and Information to be Executed by Virtual StorageApparatus>

FIG. 31, in which the same reference numeral is given to the componentscorresponding to those illustrated in FIG. 6, show the software programsto be executed by the virtual storage apparatuses 1000L, 1000R, andinformation to be used by these programs.

In FIG. 31, the AOU I/O processing program 31010 is a program forprocessing I/O request received by the virtual storage apparatuses1000L, 1000R, and contains a part of the functions of the I/O processingprogram 6020 (FIG. 6) in the first to fourteenth embodiments.

The AOU management program 31020 is a program for configurationconcerning the AOU function and executing Deduplication processingdescribed later. Further, the AOU address mapping information 31030 isinformation concerning the correspondence of the AOU volumes 29010L,29010R and the pool volume areas. Further, the AOU pool managementinformation 31040 is information for managing the aggregate of poolvolumes (pool) to be used by the AOU function.

<2.1. AOU Address Mapping Information>

FIG. 35 shows the specific contents of the AOU address mappinginformation 31010. The virtual storage apparatuses 1000L, 1000R managethe storage area of data and the like, based on the identifier of thevolume provided to the host 1100, and the address of the an area(segment) into which the address space of the AOU volume is divided fromthe top by a predetermined size (segment size). Incidentally, thissegment size is a value to be configured at the time of defining thepool.

In FIG. 35, the “AOU volume identifier” and the “address space”respectively show the identifier of the AOU volumes 29010L, 29010Rcontaining the corresponding segment, and the address of such segment inthe AOU volumes 29010L, 29010R. Further, the pool ID shows theidentifier of the pool to allocate an area to the AOU volume 29010L,29010R.

The “COW (Copy On Write) flag” is a flag showing it is necessary tostore the corresponding write data in a separately allocated pool volumearea if a write request to such segment arrives. This flag is sometimesturned “ON” showing that the write data needs to be stored in anotherpool volume with different segments being associated to the area of thesame pool volume.

The “pool volume area identifier” is identifying information showing theidentifier of the pool volume area actually storing the data to bestored in the segment. This identifying information, for instance, isconstituted of the following information.

(1) The identifier and address range of the internal volume when usingan area of an internal volume of the virtual storage apparatuses 1000L,1000R.

(2) Information for identifying the apparatus such as the port name orcommunication destination, information for identifying the volume insidean apparatus such as a LUN, and address range when including an area ofa volume in the storage apparatus 1500L.

(3) NULL in the case of an unallocated area.

The “takeover area” is information showing whether the pool volumes, inwhich an identifier is indicated in the corresponding “pool volume areaidentifier” column, are being managed by both the primary and secondaryvirtual storage apparatuses 1000L, 1000R (“Yes” when managed by both and“No” when not managed by both).

The “pair AOU volume identifier” retains an identifier of the AOUvolumes 29010L, 29010R forming a pair with the volume specified with thecorresponding AOU volume identifier. As this identifier, a combinationof an identifier of the corresponding virtual storage apparatuses 1000L,1000R and an identifier of the corresponding AOU volumes 29010L, 29010Ris used.

Incidentally, as described above, one reason that the AOU areas aremanaged in segment units is because the I/O performance will deterioratesince the management information of the AOU address mapping information31030 will become too large when managed in block units.

<2.2. AOU Pool Management Information>

FIG. 36 shows the specific configuration of the AOU pool managementinformation 31040. The AOU pool management information 31040 retains thefollowing information for each pool.

(1) Segment size

(2) List of volumes (pool volumes) allocated to the pool

(3) List of unallocated areas among the pool volume areas

(4) Unused capacity

(5) Threshold value for issuing an alert that the capacity isinsufficient

(6) Identifier of the virtual storage apparatus set as the opponent ofthe pool pair and pool ID in the apparatus. Incidentally, “pool pair”will be described later.

<3. Initialization>

Initialization of this embodiment is performed according to thefollowing routine.

1. Initialization of pool

2. Creation of AOU volume

3. Association of AOU volumes

4. Configuration of synchronous remote copy

The details are now explained. Incidentally, although there are casesbelow where the processing subject of certain processes is explained asthe “management host” or “program,” in reality, it goes without sayingthat the processor in the management host executes the correspondingprocessing based on a program stored in the corresponding memory of themanagement host, and the processor 1011 in the corresponding virtualstorage apparatuses 1000L, 1000R executes the corresponding processingbased on that program.

<3.1. Initialization of Pool>

Initialization is performed according to the following routine.

(Step 1) Based on a command from the management host 1200, the AOUmanagement program 31020 to be executed by one of the virtual storageapparatuses 1000L and 1000R creates a pool. Here, this command containsa segment size. Further, during the process of creating the pool, theAOU management program 31020 creates an entry of the AOU pool managementinformation 31040 containing the pool ID.

(Step 2) Based on processing similar to Step 1, the other virtualstorage apparatus 1000R or 1000L also creates a pool.

(Step 3) The management host 1200 issues to both virtual storageapparatuses 1000L and 1000R commands for configuring the pairs createdat Step 1 and Step 2 as a pool pair. This command contains a set of theID of the pools to become a pool pair and the Identifier of the virtualstorage apparatuses 1000L and 1000R to provide the pools. The AOUmanagement program 31020 that received the command communicates with theAOU management program 31020 of the virtual storage apparatuses 1000L,1000R of the opponent to become the pool pair, and configures thesepools as a pool pair if it is confirmed that the segment size set toboth pools is equal and that both pools have not yet been set as a poolpair. Incidentally, the AOU management program 31020 registers theidentifier of opponent's pool ID in the AOU pool management information31040 upon setting the foregoing pairs as a pool pair.

(Step 4) The management host 1200 issues a pool volume creation commandto one of the virtual storages 1000L and 1000R. Incidentally, thiscommand contains the identifier of the volume defined in the virtualstorage apparatuses 1000L and 1000R. The AOU management program 31020 ofthe virtual storage apparatus 1000L, 1000R that received the commandchanges the attribute of the designated volume to a pool volume, andadds the identifier of the designated volume to the pool volume list ofthe AOU pool management information 31040.

(Step 5) The management host 1200 issues the same command as at Step 3to the other virtual storage apparatus 1000R, 1000L. The other virtualstorage apparatus 1000R, 1000L that received the command performs thesame processing as at Step 3.

Incidentally, when the administrator determines that the internal volumeof the virtual storage apparatus 1000 will not be used in the AOU, Step4 and Step 5 may be omitted.

(Step 6) The management host 1200 issues to one of the virtual storageapparatuses 1000L, 1000R a command for configuring the volume of thestorage apparatus 1500L as a pool volume. Incidentally, to facilitateunderstanding, in the ensuing explanation, let it be assumed that thecommand destination is the virtual storage apparatus 1000L, and thevirtual storage apparatus 1000 forming a pair with the commanddestination is the virtual storage apparatus 1000R. Nevertheless, thismay also be of the opposite relationship. Here, the command includesinformation for identifying the storage apparatus 1500L and the volume,and information showing that the volume the opponent's virtual storageapparatus 1000R in which the volumes are forming a pool pair will takeover. The AOU management program 31020 of the virtual storage apparatus1000L that received the command will perform the following coordinationwith the pair virtual storage apparatus 1000R.

(A) By issuing a read request (or a write request) to the volume of thestorage apparatus 1500L contained in the command, the virtual storageapparatus 1000L that received the command confirms that both the storageapparatus 1500L and the volume exist, and whether such volume isaccessible. If the storage apparatus 1500L or the volume does not exist,or if it is not possible to access the volume, the virtual storageapparatus 1000L returns an error to the management host 1200, andproceeds to the subsequent step if the volume is accessible.Incidentally, the error includes information showing that the storageapparatus 1500L was inaccessible, and this information may be displayedon the management host 1200.

(B) The virtual storage apparatus 1000L that received the commandtransfers the pool volume creation command to the paired virtual storageapparatus 1000R. Incidentally, this command contains information foridentifying the target volume contained in the command from themanagement host 1200, and information showing that the volume is beingmanaged by both pools belonging to the pool pair. Incidentally, thetransfer destination of the pool volume creation command can bespecified by referring to the “identifying information of pool pair” inthe AOU pool management information 31040.

(C) When the virtual storage apparatus 1000R receives the command of(B), it confirms that the volume in the storage apparatus 1500L isaccessible by performing the same processing as (A). If the volume isaccessible, the virtual storage apparatus 1000R adds the volume to thepool volume list of the AOU pool management information 31040 togetherwith the information showing that the volume is being commonly managed,and returns the result to the virtual storage apparatus 1000L thatreceived the foregoing command. Meanwhile, if the volume wasinaccessible as a result of the foregoing confirmation, the virtualstorage apparatus 1000R adds information showing that the virtualstorage apparatus 1000R could not access the storage apparatus 1500L andreturns a result signifying the unsuccessful access.

(D) If the result of access to the volume was unsuccessful, the virtualstorage apparatus 1000L that received the command that received theforegoing result transfers the reason as well as the result to themanagement host 1200, and ends the sequential processing. Meanwhile, ifthe result of access to the volume was successful, the virtual storageapparatus 1000L adds this result to the pool volume list of the AOU poolmanagement information 31040 together with the information showing thatthe volume is a volume to be commonly managed, transfers a resultsignifying the successful access to the management host 1200, andthereby ends this processing.

Incidentally, when adding the volume to the pool volume list at (C) and(D), the AOU management program 21020 updates the unused capacity storedin the corresponding “unused capacity” column to a value obtained byadding the capacity of the added volume, and adds the area of thisvolume to the empty area list. Further, the processing at Step 5 mayalso issue commands separately from the management host 1200 to thevirtual storage apparatuses 1000L and 1000R.

(Step 7) The management host 1200 transfers a command for configuringthe value of a capacity warning to both virtual storage apparatuses1000L, 1000R, and the virtual storage apparatuses 1000L, 1000R thatreceived the command respectively set the value contained in the commandin the AOU pool management information 31040.

<3.2. Creation of AOU Volume>

Creation of the AOU volumes 29010L, 29010R is conducted by issuingcommands to the respective virtual storage apparatuses 1000L, 1000R. Theroutine is described below.

(Step 1) The management host 1200 transfers the AOU volume creationcommand accompanying the volume capacity and the pool ID to therespective virtual storage apparatuses 1000L, 1000R.

(Step 2) The virtual storage apparatus 1000L that received the commandcreates AOU address mapping information 31030L concerning a new AOUvolume 29010L. Here, for all segments, “NO” is set in the corresponding“COW flag” and the “takeover area”, and “NULL” is set to the “poolvolume area identifier.” The virtual storage apparatus 1000L thereafterreturns a creation complete reply.

(Step 3) Similarly, the virtual storage apparatus 1000R that receivedthe command creates AOU address mapping information 31030R concerning anew AOU volume 29010R. Details concerning the creation are the same asStep 2.

Incidentally, the AOU volumes 29010L, 29010R may be created by issuing acommand to one of the virtual storage apparatuses 1000L, 1000R from themanagement host 1200, and the virtual storage apparatus 1000L, 1000Rthat received the command re-issuing the command to the paired virtualstorage apparatus 1000R, 1000L. Incidentally, the port name and the LUNunder the control of the port name, designated by the administrator, maybe allocated to the AOU volumes 29010L, 29010R by including the portname and LUN in the volume creation command. Further, the port name andthe LUN may be allocated after the creation command of the AOU volumes29010L, 29010R.

<3.3. Association of AOU Volumes>

The AOU volumes 29010L and 29010R, each created in the respectivevirtual storage apparatuses 1000L, 1000R, are associated to each other.Thus, the management host 1200 transfers an association commandcontaining the identifier of the two AOU volumes 29010L and 29010R tothe virtual storage apparatus 1000L or 1000R. The virtual storageapparatus 1000L or 1000R that received the command register the AOUvolumes 29010L and 29010R forming a pair in the corresponding “AOUvolume identifier” column of the AOU address mapping information 31030.This command is given to the respective virtual storage apparatuses1000L and 1000R in order to associate the AOU volumes 29010L and 29010R,as described in the other embodiments, this may also be realized by oneof the virtual storage apparatus 1000L or 1000R transferring the commandto the other virtual storage apparatus 1000R or 1000L.

Incidentally, upon sending the foregoing association command, byconfirming the existence of the AOU volume 29010L and 29010R containedin the command, and that one of the AOU volumes 29010L and 29010R iscreated from one of the pools of the pool pair and the other AOU volume29010R or 29010L is created from the other pool of the pool pair,implementation of the pool management can be simplified. Further, thisassociation can be conducted pursuant to the creation of the AOU volumes29010L, 29010R and setting of the synchronous remote copy.

<3.4. Setting of Synchronous Remote Copy>

In the synchronous remote copy expressed above, although it wasnecessary to copy all areas of the volume in the Initial-Copying status,in this embodiment, formation copy is performed according to thefollowing routine. Further, in order to facilitate the understanding, inthe ensuing explanation, let it be assumed that the primary virtualstorage apparatus 1000 is the virtual storage apparatus 1000L, and thesecondary virtual storage apparatus 1000 is the virtual storageapparatus 1000R.

(Step 1) The virtual storage apparatus 1000L to become the copy source(in other words, primary system for the volume) assigns the top segmentof the AOU volume 29010L to the variable i.

(Step 2) The virtual storage apparatus 1000L of the copy source confirmsthe “takeover area” and “pool volume area identifier” of the segment iin the AOU address mapping information 31030, and performs the followingprocessing.

(A) If the “takeover area” is “NO,” data of the segment i is copiedaccording to the normal creation copy. This is because of the area ofthe pool volume in the virtual storage apparatus 1000L, copy must beperformed to secure the redundancy.

(B) When the “takeover area” is “YES,” dirty data in the cache memorynot shown in the virtual storage apparatus 1000L related to the segmenti is destaged, or copied to the cache area of the virtual storageapparatus 1000R of the copy destination (in other words, the secondarysystem for the volume) in the creation copy. This is because, excludingthe data in the cache memory, data is outside the primary virtualstorage apparatus 1000L, or, by moving the data in the cache memoryoutside the virtual storage 1000L, no data will be lost even if thefunction of the primary virtual storage apparatus 1000L is stopped.

(C) When the “pool volume area identifier” is “NULL,” copy is notperformed to the segment i since areas are not allocated to either theprimary or secondary system.

(Step 3) When the segment i is the last segment, the virtual storageapparatus 1000L of the copy source ends the creation copy and changesthe pair status to a Duplex status, and, when it is not the lastsegment, it sets the subsequent segment to the variable i and returns toStep 1.

Incidentally, the foregoing processing may be used in theresynchronization processing between the virtual storage apparatuses1000L, 1000R, or may be used in the processing after the function of oneof the virtual storage apparatuses 1000L, 1000R is stopped andrecovered.

<4. I/O Request Processing>

The I/O request processing of this embodiment is explained below.

<4.1. Write Request Processing>

FIG. 32 is a flowchart showing the processing contents to be executed bythe AOU I/O processing program 31010 upon receiving a write request.Incidentally, in the foregoing explanation, although an explanation wasnot provided using separate flowcharts regarding the command and writedata configuring the write request, since there are cases in thisprocessing were certain areas of the write request target are allocatedwhile the other areas are unallocated, the processing is explained indetail with reference to the flowchart.

(S32001) The AOU I/O processing program 31010 receives a write commandconstituting the write request. This write command contains the address(location) and the data length.

(S32100) The AOU I/O processing program 31010 executes allocationprocessing based on the received write command. By executing thisprocessing, the AOU I/O processing program 31010 checks whether an areaof the pool volume is allocated for each segment, and allocates an areaof the pool volume to segments to which an area of the pool volume isunallocated, or segments allocated with an area shared with othersegments in which the “COW flag” is “ON” (segments that need to bewritten during writing other than the shared area). Further, the AOU I/Oprocessing program 31010 reflects the allocation results of the poolvolume area in the AOU address mapping information 31030.

(S32003) The AOU I/O processing program 31010 confirms the attribute ofthe AOU volume 29010R, 29010L, and executes S32004 if the AOU volume29010R, 29010L is a copy source volume, or otherwise executes S32005.

(S32004) The AOU I/O processing program 31010 calls the remote copyprogram 6010 and thereby transfers the command of the synchronous remotecopy to the virtual storage apparatus (secondary virtual storageapparatus) 1000R having the copy destination volume.

(S32005) The AOU I/O processing program 31010 receives the write data(whole or a part) constituting the write request corresponding toS32001.

(S32006) The AOU I/O processing program 31010 confirms the attribute ofthe AOU volume 29010R, 29010L, and executes S32007 if the AOU volume29010R, 29010L is a copy source volume, and otherwise executes S32008.

(S32007) The AOU I/O processing program 31010 calls the remote copyprogram 6010 and thereby transfers the write data to the virtual storageapparatus (secondary virtual storage apparatus) 1000R having the copydestination volume.

(S32008) The AOU I/O processing program 31010 seeks the area of the poolvolume actually storing the write data from the address in the AOUvolumes 29010R, 29010L based on the AOU address mapping information31030, and stores and manages the write data related to the sought areain the cache memory.

(S32009) The AOU I/O processing program 31010 determines whether thereis a continuation of the write data reception, and re-executes S32005when there is a continuation.

(S32010) The AOU I/O processing program 31010 transfers the writecompletion reply to the primary virtual storage apparatus 1000L or thehost 1100, and ends this write request processing.

Incidentally, the secondary virtual storage apparatus 1000R handles thereception of the command of the synchronous remote copy as with thereception of the write command from the host 1100. Similarly, thevirtual storage apparatus 1000R handles the reception of data based onthe data transfer of the synchronous remote copy as with the receptionof write data from the host 1100. This concludes the explanation on thewrite request processing in the secondary virtual storage apparatus1000R.

<4.1.1. Allocation Processing>

The allocation processing illustrated in FIG. 32 is now explained.

(S32101) The AOU I/O processing program 31010 divides the write range(in other words, write address and data length) designated in the writecommand into segments.

(S32102) The AOU I/O processing program 31010 assigns the first segmentamong the plurality of segments, created by the divisioning, to thevariable i.

(S32103) The AOU I/O processing program 31010 determines the allocationstatus of the segment i and whether COW (Copy On Write) is necessary.Incidentally, the AOU address mapping information 31030 is used in thisdetermination. As a result of the foregoing determination, the AOU I/Oprocessing program 31010 executes S32105 if the allocation of the poolvolume area is not required, and executes S32104 if the pool volume areais unallocated or, if the COW flag is ON even if it is allocated (forinstance, when the segment and area of other AOU volumes 29010R, 29010Lare being shared).

(S32104) The AOU I/O processing program 31010 searches for an unusedarea from the pool volume areas to allocate such unused area to thesegment i, and then registers the searched area in the “pool volume areaidentifier” of the AOU address mapping information 31030. Incidentally,if an unused area cannot be found, the AOU I/O processing program 31010transfers a reply indicating that the write command was unsuccessful,and thereby ends this allocation processing.

Incidentally, upon transferring the failure reply, some kind of errormessage can be returned together with the failure reply, and informationindicating that the cause of the failure reply is the insufficient poolcapacity. Further, if allocating the area in a case where the “COW flag”is ON, the AOU I/O processing program 31010 may copy data from the oldarea (shared area) to the allocated area upon allocating such area.Nevertheless, when the overall segment i is a write target, this datacopy can be omitted. Further, pursuant to this area allocation, the AOUI/O processing program 31010 may edit the empty area list of the AOUpool management information, and delete the unused capacity.

Further, the AOU I/O processing program 31010 transfers the area in theallocated pool volume and information of the segment of the AOU volume29010L allocated with the area to the secondary virtual storageapparatus 1000R. Incidentally, the allocated information can also betransferred together with the synchronous remote copy command.

(S32105) The AOU I/O processing program 31010 confirms the existence ofa subsequent segment, executes S32106 if such subsequent segment exists,or ends this processing if it does not exist and then returns to thewrite request processing.

(S32106) The AOU I/O processing program 31010 assigns the subsequentsegment to the variable i.

With the foregoing processing, the virtual storage apparatus 1000Lconfirms the allocation status for each segment, and allocates the poolvolume area to the segment if necessary.

<4.1.2. Allocation Method of Secondary Pool Volume Area>

The pool volume area allocation step (S32104) of the secondary virtualstorage apparatus 1000R allocates an area to the segment according tothe following method based on the allocation information received fromthe primary virtual storage apparatus 1000L.

(A) If the primary virtual storage apparatus 1000L allocated an areafrom the pool volume of the shared storage apparatus (in other words,the storage apparatus 1500L), the secondary virtual storage apparatus1000R sets “YES” in the “takeover area” of the corresponding segment inthe AOU address mapping information 31030, and sets the “pool volumearea identifier” in the received area identifier. Thereby, allocation ofthe pool volume area concerning the shared storage apparatus 1500L willbe handled the same in both the primary and secondary systems.

(B) If the primary virtual storage apparatus 1000L allocated an areafrom the volume in the virtual storage apparatus 1000R, the secondaryvirtual storage apparatus 1000R searches for an empty area of theinternal volume to be allocated to the corresponding segment. As aresult, the secondary virtual storage apparatus 1000R sets “NO” in the“takeover area” of the segment in the AOU address mapping information31030, and sets an area of the internal volume in the “pool volume areaidentifier.” Thereby, the segment allocated with the area of theinternal volume of the by the primary virtual storage apparatus 1000Lcan also be allocated with the internal volume in the secondary virtualstorage apparatus 1000R.

<4.2. Read Request Processing>

FIG. 33 is a flowchart showing the processing contents to be executed bythe AOU I/O processing program 31010 upon receiving a read request. Theprocessing contents are now explained with reference to the flowchart.

(S33001) The AOU I/O processing program 31010 receives a read commandconstituting the read request. This received read command contains theaddress (location) and the data length.

(S33002) The AOU I/O processing program 31010 divides the read range (inother words, write address and data length) designated in the readcommand into segments.

(S33003) The AOU I/O processing program 31010 assigns the first segmentamong the plurality of segments, created by the divisioning, to thevariable i.

(S33004) The AOU I/O processing program 31010 determines whether a poolvolume area is allocated to the segment i. Incidentally, the AOU addressmapping information 31030 is used in this determination. As a result ofthe foregoing determination, the AOU I/O processing program 31010executes S33006 if the pool volume area is allocated, or executes S33005if the pool volume area is unallocated.

(S33005) The AOU I/O processing program 31010 allocates a cache area forthe segment in the cache memory of the virtual storage apparatus 1000L,1000R, initializes the allocated cache area with zero, and transfers thezero data to the host 1100.

(S33006) The AOU I/O processing program 31010 transfers the data storedin the allocated pool volume area. Incidentally, if the pool volume areaalready exists in the cache area (staged), it transfers the data fromthe cache area, or performs staging and thereafter transfers such dataif it does not exist in the cache area.

(S33008) The AOU I/O processing program 31010 determines whether thereis a subsequent segment, executes S33009 if such a subsequent segmentexists, or executes S33010 if it does not exist.

(S33009) The AOU I/O processing program 31010 sets the subsequentsegment to the variable i, and executes S33004 once again.

(S33010) The AOU I/O processing program 31010 transfers the readcompletion reply to the host 1100, and thereby ends this processing.

Incidentally, in order to simplify the processing, the virtual storageapparatus 1000L may store a predetermined value (zero) in a certain areaof the pool volume, and transfer the data stored in the area to theunallocated area of the AOU volumes 29010R, 29010L through reading.

<4.3. AOU Destaging Processing>

FIG. 34 is a flowchart showing the processing contents of destagingprocessing to be executed by the AOU I/O processing program 31010. Thedestaging processing is now explained with reference to the flowchart.

(S34001) The AOU I/O processing program 31010 decides the data in thecache memory to be destaged based on the cache algorithm. Incidentally,the cache algorithm generally uses LRU (Less Recently Used) algorithm todecide the target dirty data, but other algorithms may also be used.

(S34002) The AOU I/O processing program 31010 determines whether thedestaging target data corresponds to the volume of the shared storageapparatus (in other words, the storage apparatus 1500L), executes S34003if they correspond, or executes S34004 if they do not correspond.

(S34003) The AOU I/O processing program 31010 executes destagingprocessing, and thereafter ends the sequential processing. Incidentally,destaging processing is performed as with the other embodiments.

(S34004) The AOU I/O processing program 31010 determines the volumeattribute of the volume storing the destaging target data, executesS34005 if the volume is a copy source volume, executes S34007 if thevolume is a copy destination volume, or otherwise executes S34003.

(S34005) The AOU I/O processing program 31010 executes destagingprocessing.

(S34006) The AOU I/O processing program 31010 transfers the RC destagepermission command of the destaged data to the secondary virtual storageapparatus 1000R, and thereby ends this processing.

(S34007) The AOU I/O processing program 31010 confirms whether the RCdestage permission flag is ON, re-executes S34001 to re-select separatedestaging target data if it is OFF. Incidentally, the RC destagepermission flag is set to OFF at the time the write data is stored orupdated in the cache memory based on synchronous remote copy, and is setto ON upon receiving the command sent at S34006.

(S34008) The AOU I/O processing program 31010 executes destagingprocessing, and thereby ends this processing.

The following cache control is realized with this algorithm.

(A) Cache data that is not for the shared storage apparatus, i.e., inwhich destaging does not have to be coordinated in the primary andsecondary virtual storage apparatuses 1000L, 1000R is destagedindependently in both systems.

(B) Cache data of the secondary virtual storage apparatus 1000R isdestaged based on a message sent after the destaging processing in theprimary virtual storage apparatus 1000.

Incidentally, the staging processing is performed as in the first tofourteenth embodiments. Furthermore, Instead of the destaging at S34008,the AOU I/O processing program 31010 may discard the cache data withoutdestaging.

<4.3.1. RC Destage Permission Command>

For the transfer of the RC destage permission command, the command maybe sent asynchronously. Nevertheless, the primary and secondary virtualstorage apparatuses 1000L, 1000R may invalidate the command unreflectedin the RC destage flag by being triggered with remote copy.

<4.4. Monitoring of Empty Area of Pool>

The AOU management program 31020 periodically monitors the empty area ofthe respective pools, and sends a message to the management host 1200 ifthe value falls below the threshold value set by the user. Thereby, itis possible to avoid the failure of a write request from the host 1100caused by insufficient capacity. Further, the AOU management program31020 may manage the monitoring of the empty area of the shared storageapparatus 1500L and the unshared storage apparatus respectively, orchange a the message to be transferred during such insufficient capacityrespectively.

<5. Switching When Failure in Primary Virtual Storage Apparatus>

When the function of the primary virtual storage apparatus 1000L isstopped due to a failure or the like, the host 1100 is able to ongoinglyoperate the application by performing the processing as in the otherembodiments.

Meanwhile, the host 1100 may also switch the I/O request destination tothe secondary virtual storage apparatus 1000R if the write request tothe copy source volume is unsuccessful due to insufficient capacity.This is because if the pool capacity of the secondary virtual storageapparatus 1000R is larger than the primary system, this switch willallow the ongoing processing of the application 2010 (FIG. 30) in whichan I/O request is issued in the host 1100.

Incidentally, in the foregoing case, the direction of remote copy willbe inverted as a result of switching the request destination, but theremote copy will be stopped. This is because, since the request wasunsuccessful in the old primary virtual storage apparatus 1000L due toinsufficient pool capacity during the write request, the attempt ofwriting write data into the new primary (old secondary) virtual storageapparatus 1000R based on synchronous remote copy will end in a failure.

Nevertheless, since a request (particularly read) to the old primaryvirtual storage apparatus 1000L can be continued, it is not possible todistinguish this failure from the communication path failure between thevirtual storage apparatuses 1000L, 1000R, and the host 1100 may read olddata of the old primary virtual storage 1000L.

In order to avoid this kind of situation, if the cause of the remotecopy failure is insufficient pool capacity, the issuance of a readrequest from the host 1100 to the old primary virtual storage apparatus1000L may be inhibited. Or, while it is not possible to limit the causeof the remote copy failure, it is also possible to inhibit the readingfrom the host 1100 to the secondary virtual storage apparatus 1000R or1000L, and cancel such inhibition when it is discovered that the causeis a communication path failure.

Based on the foregoing processing, the storage system of this embodimentis able to provide a storage service having an AOU function with highservice continuity. Further, the AOU function needs to refer to andchange the AOU address mapping information 31030L, 31030R for each I/Orequest, and the controller load will be higher than a normal storageI/O. Therefore, regarding a part (or half) of the volumes required bythe host 1100, one of the virtual storage apparatuses 1000L and 1000Rmay handle the reading and writing as a primary system, and, regardingthe remaining volumes, the other virtual storage apparatus 1000R or1000L may handle the reading and writing as a primary system. As aresult of adopting this kind of constitution, the controller load of theAOU function can be equalized between the virtual storage apparatuses1000L and 1000R while maintaining the availability of the storagesystem.

<6. Allocation of Pool Volume Area and Data Migration>

As described above, in this embodiment, the volume in the virtualstorage apparatuses 1000L, 1000R and the volume of the storage apparatus1500L can both be used as pool volumes. Thus, by allocating the volumein the virtual storage apparatuses 1000L, 1000R to the segment of highaccess frequency data that is already stored or will be stored, inaddition to seeking improvement of the access performance, it ispossible to avoid the bottleneck of the communication network betweenthe virtual storage apparatuses 1000L, 1000R and the storage apparatus1500L.

Nevertheless, since the pool volume area is allocated to the segmentbased on the first write request in AOU, it would be difficult toperform allocation with the virtual storage apparatuses 1000L, 1000Ralone while giving consideration to the access frequency. The followingmethods can be considered to overcome the foregoing problem.

<6.1. Method of Adding Attribute to AOU Volume>

Consider to giving an attribute concerning the access frequency at thetime of creating the AOU volumes 29010L and 29010R and then the AOU I/Oprocessing program 31010 is to allocate a pool volume area to thesegment. If the access frequency of such data to be written into thesegment is known to a certain degree, the access frequency attribute isreferred to, and the volume in the virtual storage apparatuses 1000L,1000R is allocated to the segment of the high access frequency data, andan area of the volume in the storage apparatus 1500L is allocated to thesegment of low access frequency data (backup data, for instance).

<6.2. Data Migration of Pool Volume Area>

The access frequency to the AOU volumes 29010L, 29010R is measured insegment units (or units of a plurality of segments), and data of asegment of high access frequency is migrated to the pool volume area inthe virtual storage apparatuses 1000L, 1000R. Here, pursuant to themigration of data, it is necessary to change the correspondence of theAOU volume 29010L, 29010R's segment, that is the target of themigration, from a segment in the volume of the storage apparatus 1500Lto a segment of the migration destination in the virtual storageapparatuses 1000L, 1000R. But in the AOU function, since address mappinghas already been performed in the virtual storage apparatuses 1000L,1000R, it is possible to migrate data transparently to the host 1100 byusing it without additional mapping scheme.

Incidentally, if performing data migration as in this embodiment, it isdesirable that the data of the target segment is in the pool volume ofboth the primary and secondary virtual storage apparatuses 1000L, 1000R.Nevertheless, if there are other effects (as listed below), datamigration may also be performed into a state where only a segment of oneside is allocated with the pool volume area in the virtual storageapparatuses 1000L, 1000R.

EXAMPLE 1

If one of the virtual storage apparatuses 1000L, 1000R first exhaustedthe internal pool volume, and only the shared storage apparatus 1500Lremains.

EXAMPLE 2

If the load of the read request to the AOU volume 29010L of the copysource is large, and the network performance between the primary virtualstorage apparatus 1000L and the storage apparatus 1500L is suppressed byit.

In the foregoing case, the primary virtual storage apparatus 1000Lcopies segment data from the pool volume area in the storage apparatus1500L to the area of its own pool volume, and provides the AOU volume29010L by using the area of the copy destination. The secondary virtualstorage apparatuses 1000R may also provide the AOU volume 29010R usingthe pool volume area of the storage apparatus 1500L of the copy source.Here, the secondary virtual storage apparatus 1000R may reflect thewrite data to the pool volume area of the storage apparatus 1500L.

Further, as an intermediate status of segment data migration forimproving the access performance including reading and writing, aconfiguration may also be adopted where only the primary virtual storageapparatus 1000L uses the internal pool volume area, and the secondaryvirtual storage apparatus 1000R uses the pool volume of the storageapparatus 1500L.

<7. Variation of Present Embodiment>

<7.1. Implementation of Address Mapping During Staging and DestagingProcessing>

In this embodiment described above, address mapping is performed withthe read request processing and the write request processing. While thismethod is able to return a failure reply triggered by the insufficientcapacity of the pool volume at the time of receiving the write request,since address mapping is performed for each request, there is a problemfrom the perspective of performance. As a method of overcoming thisproblem, a method of performing address mapping with staging ordestaging processing can be considered. However, in this method, sincethe pool volume area is allocated to the segment at the time ofdestaging, data loss similar to a case of the volume being blocked dueto the double block of the HDD 1030 or the like will occur. Thus, in thelatter method, processing of delaying or stopping the request processingmay be performed from the time that the unused capacity starts fallinglow.

Incidentally, the specific processing can be realized by changing theprocessing contents explained in FIG. 32 and FIG. 33 as described below.

(Writing and Destaging) Allocation processing at S32100 of FIG. 32 ismoved after the destaging processing at S34001.

(Reading and Staging) Determination of allocation and transfer of zerodata in the case of unallocation pursuant to the address mappingperformed at S33004 to S33006 of FIG. 33 are performed at staging.

Further, in order to combine both advantages, the AOU I/O processingprogram 31010 may perform mapping in the staging/destaging processing ifthe unused capacity of the pool volume is greater than the thresholdvalue, or perform mapping in the I/O request processing if the unusedcapacity falls below the threshold value.

7.2. <De-Duplication>

The AOU management program 31010 may also perform the followingprocessing referred to as De-duplication independent from the I/Orequest.

(Step 1) The AOU management program 31010 scans the data of therespective pool volume areas and searches for redundant segments.

(Step 2) When the AOU management program 31010 detects redundant datastored in the pool volume areas, it leaves one of the areas, andreleases the other remaining areas as empty areas. Then, the “poolvolume area identifier” of the segment corresponding to the areareleased in the AOU address mapping information 31030 is updated to theone remaining area, and the “COW flag” is set to “ON.”

Here, as the method of detecting redundancy, a two-step method ofcalculating the hash value of each pool volume, thereafter sequentiallycomparing such hash value with the hash value of the other areas, andcomparing the actual data if the hash values are the same may beadopted. Further, since the calculation of the hash value and thecomparison of data are high-load processing, the load can be balanced byperforming processing in the secondary virtual storage apparatus 1000R.

(17) Seventeenth Embodiment

<1. Configuration of Information System>

FIG. 37, in which the same reference numeral is given to the componentscorresponding to those illustrated in FIG. 1, shows an example of thehardware constitution of the information system according to anembodiment of the present invention.

This embodiment differs from the first embodiment in that theremote-side storage apparatuses 2550 and the local-side storageapparatuses 1000 are connected via an I/O network 1400, and the tworemote-side storage apparatuses 2550 are coupled together. Although theI/O network 1400 may be equivalent to the I/O network 1300, generally, acommunications means that has a slower communication speed than the I/Onetwork 1300 and is communicable even at a remote location is used.Further, this embodiment also differs from the first embodiment in thatthe host 11000B is also coupled to the side of the remote-side storageapparatus 2550. Generally, the host 1100B exists at a remote site, andis used upon recovering the system at the remote site 38000B (FIG. 38)when the overall host 1100P or local-side storage apparatus 1000 at thelocal site 38000P (FIG. 38) becomes unavailable.

<2. Overview of Present Embodiment>

In this embodiment, in order to further improve the availability of thestorage system, two remote-side storage apparatuses 2550 are installedat the remote site 38000B located at a site that is remote from thelocal site 38000P constituted of two local-side storage apparatuses1000. FIG. 38 shows an overview thereof. Incidentally, in the followingexplanation, the term” local” will be added as a prefix to the storageapparatus 1000 (or 2550) or the host 1100, or the programs or componentsincluded in the storage apparatus 1000 or the host 1100 to specify thatthey exist at the local site 38000P (FIG. 38), and the term “remote”will be added as a prefix to specify that they exist at the remote site(FIG. 38, 38000B).

In this overview, the storage system includes the local-side storageapparatus 1000L, the local-side storage apparatus 1000R, the remote-sidestorage apparatus 2550L, and the remote-side storage apparatus 2550R.The local-side storage apparatuses 1000L, 1000R and the remote-sidestorage apparatuses 2550L, 2550R include journal groups 38300G1 to38300G4, and one or more data volumes 38310D1 to 38310D8 and one or morejournal volumes 38320J1 to 38320J4 are defined in these journal groups38300G1 to 38300G4. Incidentally, the journal volumes 38320J1 to 38320J4are volumes for temporarily storing update data to be used in remotecopy.

After the journal groups 38300G1 to 38300G4 of one of the storageapparatuses (local-side storage apparatus 1000L, local-side storageapparatus 1000R, remote-side storage apparatus 2550L, remote-sidestorage apparatus 2550R) are defined, when data is written into the datavolumes 38310D1 to 38310D8 belonging to the journal groups 38300G1 to38300G4, the journal data containing the write data is stored in thejournal volumes 38320J1 to 38320J4 belonging to the journal groups38300G1 to 38300G4. Further, the journal data is recorded in the journalvolumes 38320J1 to 38320J4 in the order that writing processing isperformed to the data volumes 38310D1 to 38310D8 in the journal groups38300G1 to 38300G4. The structural of the journal data and the journalvolumes 38320J1 to 38320J4 will be described later in detail.

The constitution of basic copy is now explained. The data volumes38310D1, 38310D2 belonging to the journal group 38300G1 and the datavolumes 38310D3, 38310D4 belonging to the journal group 38300G2respectively configure a synchronous remote copy pair explained in thefirst to sixteenth embodiments.

Further, the data volumes 38310D1, 38310D2 belonging to the journalgroup 38300G1 and the data volumes 38310D5, 38310D6 belonging to thejournal group 38300G3 respectively configure an asynchronous remote copypair described later.

Further, the data volumes 38310D3, 38310D4 belonging to the journalgroup 38300G2 and the data volumes 38310D7, 38310D8 belonging to thejournal group 38300G4 similarly configure an asynchronous remote copypair.

Subsequently, outline of the processing in the two local-side storageapparatuses 1000L, 1000R is explained.

(A) The local-side storage apparatus 1000L receives a write l/O command(arrow 38400) for writing into the data volume 38310D1 sent from thehost 1100P to the local-side storage apparatus 1000L via the l/O network1300.

(B) When the l/O processing program 38100P receives the write l/Ocommand, it executes a JNL creation processing program 38110, which is apart of the asynchronous remote copy program 41050P (FIG. 41), accordingto the write I/O command (arrow 38410). Thereby, write data is writteninto the data volume 38310D1 (arrow 38420), and journal data createdwith the JNL creation processing program 38110 is written into thejournal volume 38320J1 (arrow 38430). The journal data is constituted ofupdate information and write data. The update information contains anupdate number. The update number is a number added by the local-sidestorage apparatus 1000L to the I/O command, and a number is added in theorder of receiving the I/O command. In addition to the update number,the update information also contains time information added by the host11000P.

(C) Further, the I/O processing program 38100P sends a remote writerequest to the local-side storage apparatus 1000R of the remote copydestination (arrow 38440). The remote write request contains a writecommand, a logical address, a volume number, a data volume, and theupdate number and time added to the corresponding journal data by thelocal-side storage apparatus 1000L at (B) above. In reality, the I/Oprocessing program 381000P sends a remote write request by executing asynchronous remote copy program 41040P (FIG. 41). Other mattersconcerning synchronous remote copy have been explained in the first tosixteenth embodiments, and the detailed explanation thereof is omitted.

The local-side storage apparatus 1000R processes the write data to betransferred as with the write I/O command sent from the host 1100P tothe local-side storage apparatus 1000L (arrow 38450, arrow 38460, arrow38470). Nevertheless, the JNL creation processing program 38110 of thelocal-side storage apparatus 1000R creates journal data (journal dataincludes write data, and information such as the write location, writelength, update number and the like required for reflecting the writedata in the remote-side storage apparatus 2550, and the details will bedescribed later) based on the information received at (C), so that thewrite data in the primary system (copy source of remote copy) and thesecondary (copy source of remote copy) corresponds to the update numberand time.

(D) In the sequential processing described above, at the point in timethe writing of write data in all data volumes 38310D1 to 38310D4 and thewriting of journal data in the journal volumes 38320J1, 38320J2 arecomplete, the local-side storage apparatus 1000L sends a write I/Ocommand completion notice to the host 1100P.

Subsequently, outline of the processing in the two remote-side storageapparatuses 2550L, 2550R is explained.

(E) The JNLRD processing program 38140 of the remote-side storageapparatuses 2550L, 2550R sends a journal data acquisition request to thecorresponding local-side storage apparatuses 1000L, 1000R via the I/Onetwork 1300. The local-side storage apparatuses 1000L, 1000R thatreceived the acquisition request send journal data to the remote-sidestorage apparatuses 2550L, 2550R. When the JNLRD processing program38140 of the remote-side storage apparatuses 2550L, 2550R receives thejournal data, it stores this in the journal volumes 38320J3, 38320J4(arrow 38480, arrow 38490).

(F) The restoration processing program 38130 reads journal data from thejournal volumes 38320J3, 38320J4 of the remote-side storage apparatuses2550L, 2550R, and writes the write data containing such journal datainto the corresponding data volumes 38310D5 to 38310D8 (arrow 26500,arrow 26510).

Subsequently, the JNLRD processing program 38140 and the restorationprocessing program 38100 of the remote-side storage apparatuses 2550L,2550R, and the JNL creation processing program 38110 and the JNLRDprocessing program 38120 of the local-side storage apparatuses 1000L,1000R execute asynchronous remote copy. Details concerning theprocessing of asynchronous remote copy and the pair status will bedescribed later.

Outline of the processing when a failure occurs in the local-sidestorage apparatus 1000L communicating with the host 11000P under anormal status in the constitution of the storage system illustrated inFIG. 38 is now explained with reference to FIG. 39.

When a failure occurs in the local-side storage apparatus 1000L, thehost 11000P switches the path to the other local-side storage apparatus1000R with the I/O path manager 5000 (FIG. 5) explained in the previousembodiments. Details concerning the path switching and failure detectionhave been explained above, and will be omitted here. Further, when afailure occurs in the local-side storage apparatus 1000L, therestoration processing program 38130 of the corresponding remote-sidestorage apparatus 2550L may be stopped.

Like this, even when one local-side storage apparatus 1000L fails, theother local-side storage apparatus 1000R is able to continue theapplication in the host 1100P. Further, the remote-side storageapparatus 2550R is able to perform asynchronous remote copy continuouslyat the remote site.

Outline of the processing in the case of a failover (in particular, whenthe function of both primary local-side storage apparatuses 1000L, 1000Rstops) for recovering the system using the remote-side host 11000B inthe constitution of the storage system illustrated in FIG. 38 is nowexplained with reference to FIG. 40. The failover processing program41100 of the remote-side host 1100B determines which one of theremote-side storage apparatuses 2550L, 2550R is retaining the latestdata. As this example, for instance, a method of comparing the restoredupdate number (or time) of the both storage apparatuses 2550L, 2550R maybe adopted.

With the remote-side storage apparatus 2550L, 2550R retaining the latestdata as the primary system, the data volumes 38310D5 to 38310D8 in therespective journal groups 38300G3, 38300G4 of the remote-side storageapparatus 2550L and the remote-side storage apparatus 2550R form asynchronous remote copy pair, and the synchronous remote copy (arrow40100) is started and the pair status becomes a Duplex status.Subsequently, the I/O path manager 5000 or the like of the host 1100Bswitches the I/O request (indicating read and/or write) destination tothe primary data volumes 38310D5 to 38310D8 of the pair. Detailsregarding the failover processing will be explained later with referenceto a flowchart.

Like this, even when the overall local site 38000P malfunctions, it ispossible to recover the processing of the host 1100P at the remote site38000B with the same redundant constitution of two apparatuses as thelocal site 38000P. Incidentally, the I/O request start sequence and thesynchronous remote copy start sequence may be reversed.

Incidentally, in order to simplify the foregoing explanation, althoughthe operation of the cache (1020 of FIG. 37 in the case of thelocal-side storage apparatuses 1000L, 1000R) in the local-side storageapparatuses 1000L, 1000R and the remote-side storage apparatuses 2550L,2550R is not explained, when storing write data in the data volumes38310D1 to 38310D8 or storing journal data in the journal volumes38320J1 to 38320J4, such write data and journal data may foremost bestored in the cache. This is because, although the installation willbecome complex, the performance will improve since the seek time of HDDand the like will not be included in the I/O performance. Nevertheless,as a variation of this embodiment, the constitution may exclude thecache when the local-side storage apparatuses 1000L, 1000R are virtualswitches. Here, the seek time of HDD and the like will not be includedin the I/O performance because the storage apparatuses (in this case,the remote-side storage apparatuses 2550L, 2550R) coupled to thelocal-side storage apparatuses 1000L, 1000R comprise the cache.

In this embodiment, the remote-side storage apparatuses 2550L, 2550Rprimarily issue the journal data acquisition request. This is to yieldthe advantage of performing efficient transfer of journal data inconsideration of the status or the like of the remote-side storageapparatuses 2550L, 2550R, but other modes (a mode of the local-sidestorage apparatuses 1000L, 1000R primarily sending journal data) mayalso be adopted if the objective is simply to conduct a data transfer.

Further, in this embodiment, by temporarily storing update data in thejournal volumes 38320J1 to 38320J4, it is possible to realize a systemcapable of continuing remote copy in response to the temporary failureor performance change of the I/O network 1400, or changes in the writefrequency and write volume from the host 1100P.

Moreover, in this embodiment, on the local side, since the local-sidestorage apparatuses 1000L, 1000R and the remote-side storage apparatuses2550L, 2550R coordinate to create journal data, and the secondaryremote-side storage apparatuses 2550L, 2550R coordinate to restore thejournal data on the remote side to the secondary volume, the processingwill become complex in comparison to cases of using only one storageapparatus. Thus, there are times when the primary system and thesecondary system are not able to perform remote copy with the sametransfer performance due to the performance deterioration of one of thesystems (for example, when a large-volume read request is issued to theprimary local-side storage apparatus 1000L, the load will increase onlyin the primary system since the read request is not transferred to thesecondary system). The journal volumes 38320J1 to 38320J4 have anadvantage in that they are able to avoid the overflow of journal data bycontinuing to perform buffering even when there is a load bias betweenthe primary system and the secondary system.

Incidentally, the journal volumes 38320J1 to 38320J4 are constituted ofan area in one or more HDDs. As a more preferable embodiment, thejournal volumes 38320J1 to 38320J4 are constituted of a partial orentire area of the RAID parity group as with the data volumes 38310D1 to38310D8. Further, the journal volumes 38320J1 to 38320J4 in thelocal-side storage apparatuses 1000L, 1000R and the remote-side storageapparatuses 2550L, 2550R may retain internal information so that areasof the RAID parity group can be used as the journal volumes 38320J1 to38320J4 based on a journal volume request specifying the data volumes38310D1 to 38310D8.

<3. Software and Data Creation in Present Embodiment>

Programs operating in the local-side storage apparatuses 1000L, 1000Rand the remote-side storage apparatuses 2550L, 2550R, and the datastructure are now explained with reference to FIG. 41 concerningportions that are different from the first to sixteenth embodiments.

<3.1. Local-Side Storage Apparatus 1000>

As shown in FIG. 41, the local-side storage apparatuses 1000 (1000L,1000R) retain, in the memory 1012 of the controller 1010, an I/Oprocessing program 38100P, an initialization program 41030P, asynchronous remote copy program 41040P, an asynchronous remote copyprogram 41050P, copy pair information 41300P, volume information 41320P,journal group information 41330P and primary/secondary information41340P.

Among the above, the I/O processing program 38100P is a program forreceiving I/O commands and requests from the host 1100P, bootingcorresponding programs, and executing read/write processing or the liketo the volumes, and the initialization program 41030P is a program forforming a synchronous remote copy pair and an asynchronous remote copypair, and executing initialization copy.

Further, in this embodiment, the synchronous remote copy program 41040Pis a program for performing transfer processing of transferring writedata together with the update number and update time. Incidentally, thepoint of referring to the request used in this transfer as a remotewrite request is as explained in the foregoing overview.

Contents of the asynchronous remote copy processing based on theasynchronous remote copy program 41050P are as explained in the thirdembodiment. The asynchronous remote copy program 41050P differs on thelocal side and the remote side. The asynchronous remote copy program41050P of the local-side storage apparatuses 1000L, 1000R is constitutedof the JNL creation processing program 38110 and the JNLRD processingprogram 38120.

The JNL creation processing program 38110 is a program for deciding thecorresponding journal groups 38300G1, 38300G2 and journal volumes38320J1, 38320J2 upon receiving a write request for writing into thedata volumes 38310D1 to 38310D4, and writes journal data into thejournal volumes 38320J1, 38320J2. Incidentally, journal data containsthe address, write data, write sequence number and the like of the dataof the volume in the write command received by the local-side storageapparatus 1000L. Details concerning the journal data will be describedlater.

The JNLRD processing program 38120 is a program for sending journal dataaccording to a request of the JNLRD processing program 38140 (describedlater) to be executed by the remote-side storage apparatuses 2550L,2550R. Details concerning the JNLRD processing based on the JNLRDprocessing program 38120 will be described later with reference to aflowchart.

The copy pair information 41300P includes the following information inaddition to the information of the previous embodiments:

(1) Identifier of the journal groups 38300G1, 38300G2 to which the datavolumes 38310D1 to 38310D4 belong; and

(2) Type of copy (“synchronous” or “asynchronous”).

The volume information 41320P includes the type of volumes (journalvolumes 38320J1, 38320J2 or data volumes 38310D1 to 38310D4) in additionto the information of the previous embodiments.

The journal group information 41330P is information for managing thejournal groups 38300G1, 38300G2, and, as shown in FIG. 47, contains thefollowing information for each of the journal groups 38300G1, 38300G2identified by the journal group number.

(1) Pair identifier list: Information storing the identifier of the copypairs belonging to the journal groups 38300G1, 38300G2.

(2) Journal volume number list: Information storing the identifier ofthe journal volumes 38320J1, 38320J2 belonging to the journal groups38300G1, 38300G2.

(3) Update number: Recorded in the update information of journal data,and used for protecting the update sequence of data in the remote-sidestorage apparatuses 2550L, 2550R, and determining from the remote sidethe remote-side storage apparatuses 2550L, 2550R having the latest datavolumes 38310D5 to 38310D8 at the time of recovery.

(4) Opponent storage apparatus number: Stores information foridentifying the storage apparatuses (local-side storage apparatuses1000L, 1000R or remote-side storage apparatuses 2550L, 2550R) of theremote copy destination.

(5) Opponent journal group number: Stores information for identifyingthe journal groups 38300G1 to 38300G4 of the remote copy destination.

(6) Oldest address of update information

(7) Latest address of update information

(8) Start address of update information transfer

(9) Oldest address of write data

(10) Latest data of write data

(11) Restored latest update number

Incidentally, (6) to (11) will be described later.

The primary/secondary information 41340P is information for managing therole of storage apparatuses (for instance, when the local-side storageapparatus 1000L is a self-storage apparatus, the local-side storageapparatus 1000R and the remote-side storage apparatuses 2550L, 2550Rcorresponding thereto) other than the self-storage apparatus configuringthe system, and contains the following information.

(1) System status (“primary” or “secondary”) and journal group number ofthe self-storage apparatus. The self-storage apparatus having theprimary/secondary information 41340P stores information showing whetherthe status is primary or secondary, and an identifier of the journalgroups in the self-storage apparatus. Incidentally, primary andsecondary represent the primary/secondary of the switch destination ofthe I/O path viewed from the hosts 1100P, 1100B. In this embodiment, thelocal-side storage apparatus 1000L and the remote-side storage apparatus2550L are “primary,” and the local-side storage apparatus 1000R and theremote-side storage apparatus 2550R are “secondary.”

(2) Apparatus number and journal group number of the opponent storageapparatus in the self site. The identification number of the opponentstorage apparatus constituted as high availability based on synchronousremote copy and the identifier of the corresponding journal groups inthe opponent storage apparatus are stored.

(3) Apparatus number and journal group number of the storage apparatusrecognized by the I/O path manager as a primary system/secondary system(hereinafter referred to as the “primary/secondary storage apparatus”)in other sites. When the storage apparatus managing the foregoinginformation exists in the local site 38000P, the identifier of thejournal group relating to the apparatus identifier of the primarystorage apparatus in the remote site 38000B is stored, and, when thestorage apparatus managing the foregoing information exists in theremote site 38000B, the identifier of the journal group relating to theapparatus identifier of the primary storage apparatus in the local site38000P is stored.

Incidentally, the primary/secondary information 41340P is changed uponreceiving a command issued pursuant to the switch of the write requestor read request destination of the I/O path manager 5000 (FIG. 5) of thehost 1100P. Incidentally, when the secondary storage apparatus 1000R isin a state of not being able to directly communicate with the hosts1100P, 1100B, the local-side storage apparatus 1000L may relay theforegoing command.

Further, the cache memory 1020 stores the write data 41210 and theupdate information 41220, which are information configuring journaldata. Details of the foregoing information will be described later.

<3.2. Remote-Side Storage Apparatus 2550>

The remote-side storage apparatuses 2500 (2550L, 2550R) retain, in thememory 1012 of the controller 1010, an I/O processing program 38100B, aninitialization program 41030B, a synchronous remote copy program 41040B,an asynchronous remote copy program 41050B, copy pair information41300B, volume information 41320B, journal group information 41330B, andprimary/secondary information 41340B.

Among the above, the I/O processing program 38100B, the initializationprogram 41030B, the synchronous remote copy program 41040B, theasynchronous remote copy program 41050B, the copy pair information41300B, the volume information 41320B, the journal group information41330B, and the primary/secondary information 41340B are the same as thecorresponding elements of the local-side storage apparatuses 1000L,1000R, and the explanation thereof is omitted.

The asynchronous remote copy program 41050B is constituted of the JNLRDprocessing program 38140, the restoration processing program 38100 andthe JNL creation processing program 38110.

The JNLRD processing program 38140 is a program for acquiring journaldata from the local-side storage apparatuses 1000L, 1000R. Specifically,a JNLRD command designating a journal group number of the journal groups38300G1, 38300G2 existing in the local-side storage apparatuses 1000L,1000R is sent to the command-target local-side storage apparatuses1000L, 1000R. Details of the processing based on the JNLRD processingprogram 38140 are now explained with reference to a flowchart.

The restoration processing program 38100 is a program for writing writedata stored in the journal volumes 38320J3, 38320J4 belonging to thejournal groups 38300G3, 38300G4 of the remote-side storage apparatuses2550L, 2550R into the corresponding data volumes 38310D5 to 38310D8belonging to the journal groups 38300G3, 38300G4. Details concerning therestoration processing based on the restoration processing program 38100will be explained later with reference to a flowchart.

Incidentally, other processing and programs not illustrated in FIG. 41and described in the other embodiments may also be included in therespective storage apparatuses.

<4. Structure of Journal Volume and Relationship with Journal GroupInformation>

Structure of the journal volumes 38320J1 to 38320J4 respectivelyretained in the local-side storage apparatuses 1000L, 1000R and theremote-side storage apparatuses 2550L, 2550R is now explained withreference to FIG. 42.

The journal volumes 38320J1 to 38320J4 are used by being divided intothe two areas of write data area and update information area.

<4.1. Update Information Area>

The update information area is an area for storing the updateinformation of journal data, and the update information oldest addressand the update information latest address of the journal groupinformation 41330P, 41330B (FIG. 41) manage the update information 41220(FIG. 41) to be retained by the local-side storage apparatuses 1000L,1000R or the remote-side storage apparatuses 2550L, 2550R in aring-buffer format. Further, the update information read start addressof the journal group information 41330P, 41330B shows the updateinformation of journal data to be transferred by the JNLRD processingprogram 38140 (FIG. 41). Thus, when the JNLRD processing program 38140receives a journal acquisition request that does not designate theacquisition target, it transfers the update information shown in theread start address.

FIG. 46 is a chart (46001) showing the information retained in theupdate information. Each type of information is explained below.

(A) The update time is information showing the time that the local-sidestorage apparatuses 1000L, 1000R received a write request.

(B) The update number is information showing the sequence relation ofthe write data containing the update information.

(C) The write address and write data length are information showing thevolume designated in the write request, address in the volume, andlength of write data (FIG. 42; 42100 and 42110).

(D) The journal volume address is information showing the journalvolumes 38320J1 to 38320J4 storing the replication of write data, andthe address in the journal volumes 38320J1 to 38320J4 (FIG. 42; 42120).

<4.2. Write Data Area>

The write data area is an area storing the write data contained in thejournal data, and the write data oldest address and the write datalatest address of the journal group information 41330P, 41330B managethe write data to be retained by the local-side storage apparatuses1000L, 1000R or the remote-side storage apparatuses 2550L, 2550R in aring-buffer format.

<4.3. When Using Plurality of Journal Volumes>

Due to the following reasons, there are cases where a plurality ofjournal volumes 38320J1 to 38320J4 belong to a certain journal group38300G1 to 38300G4.

(Reason 1) Flexibility of capacity: There may be cases the amount ofjournal data to be retained by one journal volume 38320J1 to 38320J4 istoo small.

(Reason 2) Problems in terms of performance: Since journal dataequivalent to the write request of all data volumes 38310D1 to 38310D8in the journal groups 38300G1 to 38300G4 is created in the journalvolumes 38320J1 to 38320J4, this easily becomes a bottleneck.

Thus, when the local-side storage apparatuses 1000L, 1000R or theremote-side storage apparatuses 2550L, 2550R use a plurality of journalvolumes 38320J1 to 38320J4, the following modes may be adopted in viewof the foregoing problems.

(Mode 1) A mode known as concatenate of adding a new journal volumebehind a certain journal volume 38320J1 to 38320J4. The various types ofinformation managing the journal volumes 38320J1 to 38320J4 will conductsuch management under the addressing rule allocated after being added.

(Mode 2) A striping mode of switching the journal volumes 38320J1 to38320J4 of the storage destination via rotation for each data volume orcreation count of predetermined journal data. As with mode 1, varioustypes of information managing the journal volumes 38320J1 to 38320J4will conduct such management under the addressing rule allocated afterbeing added.

Further, although the update area of the journal volumes 38320J1 to38320J4 will be used by being divided into an update information areaand a write data area, a different mode may be adopted for eachdifferent area.

<4.4. Variation>

The journal volumes 38320J1 to 38320J4 may also collectively retain theupdate information and write data without retaining the updateinformation and write data in separate areas.

<5. Pair Status of Asynchronous Remote Copy>

The pair status of asynchronous remote copy is now explained withreference to FIG. 43. Explanation of the same status as synchronousremote copy explained in the previous embodiments is omitted, and onlythe different points will be explained.

(1) Duplex status (43010)

In asynchronous remote copy, since the copy from the primary volume tothe secondary volume is conducted asynchronously as a result of goingthrough the journal data, even in a Duplex status, the secondary volumewill follow slightly behind the primary volume.

(2) Suspending status (43020)

The Suspending status is an intermediate status during the transitionfrom the Duplex status to a Suspend status.

<6. Initialization of System>

The initialization processing of this system using the initializationprograms 41030P, 41030B for setting the journal group information41330P, 41330B and setting the copy pair information 41300P, 41300B inorder to start the operation of the system is now explained withreference to FIG. 44.

The initialization programs 41030P, 41030B are executed as a result ofbeing triggered by the reception of an initialization command from thehosts 1100P, 1100B or the management host 1200 (FIG. 37). Further, theinitialization command contains the following information.

(A) Apparatus number of the primary local-side storage apparatus 1000Land the secondary storage apparatus 1000R

(B) Apparatus number of the remote-side primary storage apparatus 2550Land the remote-side storage apparatus 2550R

(C) Identifier of the volume to be used as the journal volume 38320J1among the volumes in the primary local-side storage apparatus 1000L

(D) Identifier of the volume to be used as the journal volume 38320J2among the volumes in the local-side secondary storage apparatus 1000R

(E) Identifier of the volume to be used as the journal volume 38320J3among the volumes in the remote-side primary storage apparatus 2550L

(F) Identifier of the volume to be used in the journal volume 38320J4among the volumes in the secondary remote-side storage apparatus 2550R

(G) Identifier of the volume of the primary local-side storage apparatus1000L and the volume of the secondary storage apparatus 1000R to be setas a synchronous remote copy pair

(H) Identifier of the volume of the primary local-side storage apparatus1000L and the volume of the remote-side local-side storage apparatus1000R to be set as an asynchronous remote copy

(I) Identifier of the volume of the primary local-side storage apparatus1000L and the volume of the remote-side primary storage apparatus 2550Lto be set as an asynchronous remote copy pair

Incidentally, these commands may be issued in parts. The routineillustrated in FIG. 44 is now explained in detail.

(S44001) When the initialization programs 41030P, 41030B are started,creation processing of the journal groups 38320J1 to 38320J4 isperformed. A more specific routine of the creation processing of thejournal groups 38320J1 to 38320J4 is described below.

(A) The local-side storage apparatus 1000L of the local-side primarysystem uses the information contained in the initialization command todefine the journal group 38300G1, and sets the volume designated by thejournal group 38300G1 to be used as the journal volume 38320J1.

Incidentally, when this initialization command to the local-side storageapparatus 1000L is not directly given from the host 1100P to thelocal-side storage apparatus 1000L of the local-side primary system, itis relayed by storage apparatus (for instance, the local-side storageapparatus 1000R of the local-side secondary system) that received theinitialization command, and, by way of compensation, a journal groupidentifier is returned from the

local-side storage apparatus 1000L to the local-side storage apparatus1000R that conducted the relay. Further, pursuant to the setting of thejournal volume 38300G1, the update number of the journal groupinformation 41330P is set to 0, and the update information oldestaddress, update information latest address and update informationtransfer start address are also set to the top address of the updateinformation area. Similarly, the write data oldest address and the writedata latest address are set as the top address of the write data area.

(B) Similarly processing is performed to the remote-side primary storageapparatus 2550L.

(C) Similarly processing is performed to the local-side secondarystorage apparatus 1000R and the secondary remote-side storage apparatus2550B.

(D) The storage apparatus (local-side storage apparatus 1000L, 1000R orremote-side storage apparatus 2550L, 2550R) that relayed theinitialization command communicates with each of the other storageapparatuses (local-side storage apparatuses 1000L, 1000R or remote-sidestorage apparatuses 2550L, 2550R) so as to issue a command for settingthe necessary values as the opponent storage apparatus number and theopponent journal group number to be included in the journal groupinformation 41330P, 41330B. Each of the other storage apparatuses(local-side storage apparatuses 1000L, 1000R or remote-side storageapparatuses 2550L, 2550R) that received this command sets the variousvalues to realize the copy topology illustrated in FIG. 38.

(S44002) The initialization programs 41030P, 41030B register a pair. Amore detailed operation of the pair creation processing is explainedbelow.

(A: Pair setting of synchronous remote copy) The pair setting ofsynchronous remote copy of forming a pair with the data volumes 38310D1and 38310D3, and a pair with the data volumes 38310D2 and 38310D4 isperformed. Specifically, the storage apparatus number of the local-sidestorage apparatus 1000L and the volume number of the data volume 38310D1are set as the storage apparatus number and the volume number of thestorage apparatus having a copy source volume, and the storage apparatusnumber of the local-side storage apparatus 1000R of the volume number ofthe data volume 38310D3 are set as the storage apparatus number and thevolume number of the storage apparatus having a copy destination volumeof the copy pair information 41300P of the local-side storage apparatus1000L. Further, “Simplex” is set as the pair status of the foregoingcopy pair. Further, “synchronous” is set as the type of copy of the pairstatus.

(B: First pair setting of asynchronous remote copy) The storageapparatus number of the local-side storage apparatus 1000L and thevolume number of the data volume D1 are set as the storage apparatusnumber and the volume number of the storage apparatus having a copysource volume, and the storage apparatus number of the local-sidestorage apparatus 1000R of the volume number of the data volume D5 areset as the storage apparatus number and the volume number of the storageapparatus having a copy destination volume of the copy pair information41300P of the local-side storage apparatus 1000L. Further, “Simplex” isset as the pair status of the foregoing copy pair. Further,“asynchronous” is set as the type of copy of the pair status.Subsequently, the same contents as the copy pair information 41300P ofthe local-side storage apparatus 1000L are set as the copy pairinformation 41300B of the remote-side storage apparatus 2550.

Further, a pair number as a number for identifying the respective copypair information 41300P, 41300B is added to the pair identifier list ofthe journal group information 41330P, 41330B of the local-side storageapparatus 1000L and the remote-side storage apparatus 2550L.

Moreover, when there are a plurality of data volumes to be added to thejournal groups 38300G1, 38300G3, the copy pair information 41300P,41300B and the journal group information 41330P, 41330B are similarlyset as described above for each data volume.

(C: Second pair setting of asynchronous remote copy) The same paircreation processing as (B) above is performed to the local-side storageapparatus 1000R of the local-side secondary system and the remote-sidestorage apparatus 2550R of the remote-side secondary system.

(S44003) The initialization programs 41030P, 41030B start the initialcopy based on the synchronous remote copy programs 41040P, 41040B.Incidentally, this initial copy is started by the relaying storageapparatus issuing a command to the storage apparatus involved insynchronous remote copy.

(S44004) The initialization programs 41030P, 41030B waits for theinitial copy at S44003 to end, and then starts the initial copy ofasynchronous remote copy from the primary local-side storage apparatus1000L to the remote-side primary storage apparatus 2550L. Incidentally,this initial copy is started by the relaying storage apparatus issuing acommand to the storage apparatus involved in asynchronous remote copy.Further, upon performing this initial copy, the remote-side primarystorage apparatus 2550L starts the JNLRD processing based on the JNLRDprocessing program 38140 and the restoration processing based on therestoration processing program 38100.

(S44005) The initialization programs 41030P, 41030B start the initialcopy of asynchronous remote copy from the local-side secondary storageapparatus 1000R to the secondary remote-side storage apparatus 2550R.Incidentally, this initial copy is started by the relaying storageapparatus issuing a command to the storage apparatus involved inasynchronous remote copy. Further, upon performing this initial copy,the secondary remote-side storage apparatus 2550R starts the JNLRDprocessing based on the JNLRD processing program 38140 and therestoration processing based on the restoration processing program38100.

(S44006) The initialization programs 41030P, 41030B change the failurestatus corresponding to the foregoing pair of the device relation tablein the host 1100P to “normal status” after both pair statuses ofasynchronous remote copy are changed to Duplex status. Further, thelocal-side storage apparatus 1000L sets “primary” as the system statusof the self-storage apparatus of the primary/secondary information41340P, the local-side storage apparatus 1000R sets “secondary” as thesystem status of the self-storage apparatus of the primary/secondaryinformation 41340B, the remote-side storage apparatus 2550L sets“primary” as the system status of the self-storage apparatus of theprimary/secondary information 41340B, and the remote-side storageapparatus 2550R sets “secondary” as the system status of theself-storage apparatus of the primary/secondary information 41340B.Further, the journal group number of each storage apparatus (local-sidestorage apparatus 1000L, 1000R or remote-side storage apparatus 2550L,2550R) is stored in the primary/secondary information 41340P, 41340B.

The routine is as described above. Incidentally, in the foregoingexplanation, although the setting for the initialization of thelocal-side storage apparatuses 1000L, 1000R and the remote-side storageapparatuses 2550L, 2550R is configured at that same time, initializationof the local-side storage apparatuses 1000L, 1000R and the remote-sidestorage apparatuses 2550L, 2550R can also be performed by the hosts1100P, 1100B making the respective settings separately through thefunction I/F or the like.

<7. Failover Processing of System>

The failover processing based on the failover processing program 41100(FIG. 40) is now explained with reference to FIG. 45.

The failover processing is processing of recovering the system with theremote host 1100B when a failure occurs in the local-side storageapparatus (local-side storage apparatuses 1000L, 1000R). The failoverprocessing is executed based on the failover processing program 41100loaded in the remote-side host 11000B (secondary host).

(S45001) Foremost, the failover processing program 41100 issues acommand of stopping the JNLRD processing to the JNLRD processing program38140 of the remote-side storage apparatus 2550R via the function I/F.The remote-side storage apparatus 2550R that received this command stopsthe JNLRD processing. Incidentally, this step may be omitted when theJNLRD processing is automatically stopped based on the asynchronousremote copy function.

(S45002) Subsequently, the failover processing program 41100 confirmsthat the two remote-side storage apparatuses 2550L, 2550R have completedthe restoration processing of the respective journal groups 38300G3,38300G4. For example, the secondary host 1100B issues a command to theremote-side storage apparatuses 2550L, 2550R via the function I/F toreturn a completion notice at the time the restoration processing basedon the restoration processing program 38100 is complete.

(S45003) Subsequently, the failover processing program 41100 selects theremote-side storage apparatus 2550L, 2550R having the latest journalgroup 38300G3, 38300G4. For example, by adding an update number of thejournal group information 41330B to the completion notice, the secondaryhost 11000B is able to select the remote-side storage apparatus 2550L,2550R having the newest update number as the remote-side storageapparatus 2550L, 2550R having the latest journal group 38300G3, 38300G4.The update number to be used in this determination be a time stamp or amarker.

(S45004) Subsequently, the failover processing program 41100 uses thefunction I/F and the like creates a synchronous remote copy pair withthe data volumes 38310D5 to 38318D8 existing in the journal groups38300G3, 38300G4 of the selected remote-side storage apparatuses 2550L,2550R as the primary system.

(S45005) Subsequently, the secondary host 1100B starts the host accessto the data volumes 38310D5 to 38318D8 existing in the selected journalgroups 38300G3, 38300G4.

(S45006) Finally, the failover processing program 41100 changes thedevice relation table of the host 1100B when the pair status of thesynchronous remote copy of the data volumes 38310D5 to 38318D8 ischanged to a “Duplex” status.

Nevertheless, when recovering remote copy with one storage system, afterstep S45003 in the foregoing flowchart, step S45004 onward can beomitted by setting the I/O path manager 5000 so that the host 1100B willonly recognize a storage apparatus having a new time stamp.

Incidentally, in order to simplify the explanation, although it wasexplained that the failover processing program 41100 is executed basedon the failover processing program 41100 loaded in the remote-side host1100B, this may also be loaded as a program that operates by thecontroller 1010 in the remote-side storage apparatuses 2550L, 2550R viathe function I/F.

<8. Details of Synchronous Remote Copy Program>

The JNLRD processing based on the JNLRD processing program 38120 is nowexplained according to FIG. 48.

The JNLRD processing is started when the local-side storage apparatuses1000L, 1000R receive a journal read request from the remote-side storageapparatuses 2550L, 2550R. Incidentally, the received journal readrequest contains the latest update number of the journal data subject torestoration processing by the remote-side storage apparatuses 2550L,2550R.

(S48008) The JNLRD processing program 38120 determines whether a retryoption accompanying information (update number, etc.) showing thejournal data to be retransferred is added to the journal read request,and proceeds to S48006 when such retry option is added (S48008; Y). Whena retry command is not added (S48008; N), the JNLRD processing program38120 proceeds to S48001.

(S48006) The JNLRD processing program 38120 seeks the update informationfrom the update sequence or the like showing the journal data to beretransferred, and identifies the write data showed in the updateinformation.

(S48007) The JNLRD processing program 38120 transfers the write data andupdate information identified at step S48006 as journal data.

(S48001) The JNLRD processing program 38120 confirms the existence of anunsent journal, and proceeds to S48005 when an unsent journal does notexist (S48001; N), and proceeds to S48002 when an unsent journal dataexists (S48001; Y). This determination is made based on whether theupdate information transfer start address and the update informationlatest address of the journal group information 41330P are the same.

(S48005) The JNLRD processing program 38120 reports “no journal” to thetarget remote-side storage apparatuses 2550L, 2550R.

(S48002) The JNLRD processing program 38120 identifies the write dataindicated in the update information shown in the update informationtransfer start address of the journal group information 41330P.

(S48003) The JNLRD processing program 38120 transfers the write data andthe update information identified at step S48002 as journal data, andchanges the update information transfer start address to the address ofthe subsequent update information.

(S48004) The JNLRD processing program 38120 waits for a reply concerningthe journal transfer from the remote-side storage apparatuses 2550L,2550R, releases the storage area of the journal volumes 38320J1, 38320J2used for the journal data upon receiving the reply, and thereby endsthis processing. Incidentally, release of the storage area of thejournal volumes 38320J1, 38320J2 can be realized by performing thefollowing update.

(A) Updating the value of the update information oldest address to avalue obtained by adding the data size of update information and thevalue of the current update information oldest address so as to indicatethe subsequent update information.

(B) Updating the value of the write data oldest address to a valueobtained by adding the data size of write data and the value of thecurrent write data oldest address so as to indicate the address of thesubsequent write data.

(C) Updating the restored latest update number of the journal groupinformation 41330P to the restored latest update number incidental tothe journal read request.

Incidentally, as a variation of this embodiment, the release of thejournal data in the local-side storage apparatuses 1000L, 1000R may alsobe performed by waiting for the transfer complete message from theremote-side storage apparatuses 2550L, 2550R, and at a timing that isimmediately after the transfer at S48003 and before the restoration iscomplete. Further, as with the foregoing embodiments, the informationcan be read from the journal volumes 38320J1, 38320J2 for transferringthe update information and write data and staged to the cache memory1020 (FIG. 37), and, when such information already exists in the cachememory 1020, staging to the cache memory 1020 may be omitted.

The journal creation processing program 38110 is now explained withreference to FIG. 49. This processing is called by the I/O processingprogram 38100P when the local-side storage apparatuses 1000L, 1000Rreceive a write request or a remote write request from the host 1100P.This processing is explained with reference to the flowchart.

(S49001) The journal creation processing program 38110 specifies thejournal groups 38300G1, 383001G2 belonging to the data volumes 38310D1to 38310D4 designated in the write request or the remote write request,and specifies the corresponding the journal volumes 38320J1, 38320J2.

(S49002) The journal creation processing program 38110 determineswhether the data volumes 38310D1 to 38310D4 designated in the writerequest or the remote write request are primary volumes of synchronousremote copy by referring to the copy pair information 41300P. When thedata volumes 38310D1 to 38310D4 are primary volumes of synchronousremote copy (S49002; Y), the journal creation processing program 38110proceeds to S49003. Further, when the data volumes 38310D1 to 38310D4are not primary volumes of synchronous remote copy (S49002; N), thejournal creation processing program 38110 proceeds to S49005.

(S49003) The journal creation processing program 38110 creates updateinformation by performing the following routine.

(A) The current time in the local-side storage apparatuses 1000L, 1000Ris set in the update time of update information. Incidentally, when themainframe host sends a write request, since the time can be associatedwith the write request, the reception time can also be set uponreceiving this kind of request.

(B) A value obtained by adding 1 to the update number of thecorresponding journal group information 41330P is set as the updatenumber of update information.

(C) The information (logical volume number, write start address, writedata length) to be contained in the write request is set as the writeaddress and write data length of update information. Incidentally,although information of the write request may be stored as is as theforegoing information, a value based on the addressing rule used in thelocal-side storage apparatuses 1000L, 1000R may also be set.

(D) A value obtained by adding 1 to the value of the foregoinginformation is set as the update number of the journal group information41330P.

(S49004) The journal creation processing program 38110 allocates astorage area of update information and write data in the journal volumes38320J1, 38320J2, and sets the top address of the write data storagearea to the journal volume address of update information created at stepS49003. Subsequently, the foregoing update information and write dataare written into the storage area allocated in the journal volumes38320J1, 38320J2. Incidentally, the following information of the journalgroup information 41330P is updated pursuant to the foregoing writing.

(A) The update information latest address is set to the storage area ofthe update information.

(B) The write data latest address is set to the storage area of thewrite data.

(S49005) The journal creation processing program 38110 proceeds toS49006 when the data volumes 38310D1 to 38310D4 are secondary volumes ofsynchronous remote copy (S49005; Y), and end the processing when thedata volumes 38310D1 to 38310D4 are not secondary volumes of synchronousremote copy (S49005; N).

(S49006) The journal creation processing program 38110 creates updateinformation from the remote write request. In creating the updateinformation, the update number contained in the remote write request isset as the update number of update information. Further, in creating theupdate information, the update time is set as the time contained in theremote write request. Elements of other update information also performthe setting of corresponding information.

Incidentally, information (for instance, update number) received withthe remote write request and used to create the update information mayalso be set in the journal group information 41330P. Incidentally, theupdate number transferred by the local-side storage apparatus 1000L ofthe primary system based on the remote write request is the same valueas the update number contained in the update information created atS49003.

The JNLRD processing is now explained with reference to FIG. 50. TheJNLRD processing is processing to be executed with the remote-sidestorage apparatuses 2550L, 2550R. The initialization program 41030Bdesignates the start of execution, and the remote-side storageapparatuses 2550L, 2550R thereafter detect a failure concerning theremote copy failure, and the JNLRD processing is repeated until acommand is received from another storage apparatus or the host 1100B.

(S50001) The JNLRD processing program 38140 sends a journal read requestto the local-side storage apparatuses 1000L, 1000R. Incidentally, whenthe JNLRD processing program 38140 is requested to resend the journaldata from another processing or the host 1100B, it sends resends suchjournal upon adding information (update number, etc.) for identifyingjournal data to be subject to a retry option and retransfer to theforegoing request. Further, the JNLRD processing contains the value ofthe restored latest update number of the journal group information41330B.

(S50002) The JNLRD processing program 38140 receives a reply from thelocal-side storage apparatuses 1000L, 1000R.

(S50003) The JNLRD processing program 38140 determines whether thecontent of the reply is “no journal,” and proceeds to S50006 when thereply is “no journal” (S50003; Y), and proceeds to S50004 when the replyis not “no journal” (S50003; N).

(S50006) The JNLRD processing program 38140 waits for a given period oftime.

(S50004) The JNLRD processing program 38140 determines whether there isshortage of an area for storing journal data in the journal volumes38320J3, 38320J4 of the self-storage apparatus, and proceeds to S50005when a journal data storage area cannot be allocated (S50004; Y), andproceeds to S50007 when a journal data storage area could be allocated(S50004; N). Incidentally, this determination can be made by referringto the following information of the journal group information 41330B.

(A) When the update information latest address is smaller than theaddress added with the update information of journal data that receivedthe update information oldest address, the storage area of updateinformation is insufficient.

(B) When the write data latest address is smaller than the address addedwith the write data of journal data that received the write data oldestaddress, the storage area of write data is insufficient.

(S50007) The JNLRD processing program 38140 discards the receivedjournal data, and waits a given period of time.

(S50005) The JNLRD processing program 38140 allocates a storage area ofupdate information and write data in the journal volumes 38320J3,38320J4, and sets the top address of the write data storage area as thejournal volume address of the update information of the received journaldata. The foregoing update information and write data are written intothe storage area allocated in the journal volumes 38320J3, 38320J4.Incidentally, the following information of the journal group information41330B is updated pursuant to the foregoing writing.

(A) The update information latest address is set to the storage area ofupdate information.

(B) The write data latest address is set to the storage area of writedata.

Incidentally, the remote-side storage apparatuses 2550L, 2550R maysimultaneously issue a plurality of journal read requests.

The restoration processing program 38100 is now explained with referenceto FIG. 51. The restoration processing program 38100 is executed by theremote-side storage apparatuses 2550L, 2550R, and the initializationprogram 41030B starts the processing.

(S51001) The restoration processing program 38100 checks whether arestoration-target journal data exists. This checking is performed byreferring to the journal group information 41330B, referring to thepointer information of the journal volumes 38320J3, 38320J4correspondingly to the respective journal groups 38300G3, 38300G4,comparing the update information oldest address and the updateinformation latest address, and determining whether the difference is 0.The restoration processing program 38100 proceeds to S51002 when arestoration-target journal data exists (S51001; Y), and waits for thearrival of a restoration-target journal data when a restoration-targetjournal data does not exist (S51001; N) since all journal data are in anapplied status (this status is sometimes referred to as “restorationprocessing complete”) (loop S51001).

(S51002) The restoration processing program 38100 reads all or a part ofone or more pieces of update information stored in the updateinformation area of the journal volumes 38320J3, 38320J4, and selectsthe write data to be reflected in the data volumes 38310D5 to 38310D8.Incidentally, this decision of write data is conducted by rearranging aplurality of pieces of update information according to the update numbersequence, and the update information in which the update number iscontinuing from the number subsequent to the update information oldestaddress (showing the latest update number of the restored journal data)of the journal group information 41330B becomes the target of selection.Incidentally, the foregoing rearranging processing is necessary totransfer the journal read requests independently from the sequence whenthey are processed in parallel.

(S51003) The restoration processing program 38100 writes the write datahaving one or more journal data decided at S51002 into the data volumes38310D5 to 38310D8. Incidentally, when there are a plurality ofselection-target journal data, this processing may also write the updatenumber sequence into the respective journal data. Further, the followingprocessing may be performed in order to speed up the foregoing writingprocess.

(A) When it is discovered that a plurality of write data have beenwritten into the same address, only the last write data is written, andthe writing of the other write data is omitted.

(B) When there are a plurality of write data in which another address isthe writing destination, the writing of such write data is performed inparallel or randomly.

Incidentally, since the data volumes 38310D5 to 38310D8 will be of aninconsistent state (write I/O sequence is not retained) in allprocessing of this step, when ending this restoration processing, it isnecessary to end all writing processing in this step to realize a statuswhere consistency is secured. Further, the restored latest update numberof the journal group information 41330B is set as the latest value amongthe update information corresponding to the plurality of write datawritten in this step.

(S51004) The restoration processing program 38100 releases the journaldata written at S51003. The method of release is the same as in theJNLRD processing, and the explanation thereof is omitted.

In this section, an asynchronous remote copy mode was explained wherethe journal data is transferred by the remote-side storage apparatuses2550L, 2550R taking lead in sending to the journal acquisition requestto the local-side storage apparatuses 1000L, 1000R. Incidentally, a modewhere the local-side storage apparatuses 1000L, 1000R take the lead asthe journal data transfer mode may also be considered. Morespecifically, the following changes or additions are made to theprevious explanation.

(A) The local-side storage apparatuses 1000L, 1000R periodically monitorthe journal volumes 38320J1, 38320J2 and transfer journal data upondiscovering such journal data.

(B) The remote-side storage apparatuses 2550L, 2550R receive the sentjournal data and thereafter store the journal data in the journalvolumes 38320J3, 38320J4, and return the latest update number of therestored journal data together with the reply of transfer complete.Further, the update number of the journal data determined by theremote-side storage apparatuses 2550L, 2550R as requiring retransfer mayby returned to the local-side storage apparatuses 1000L, 1000R togetherwith the reply.

Further, as the timing of the local-side storage apparatuses 1000L,1000R releasing the journal data, in addition to determining the timingbased on the latest update number of the restored journal data, a methodof releasing the journal data when the local-side storage apparatuses1000L, 1000R complete transferring the journal data may also beconsidered.

<9. Network Failure Between Local-Side Storage Apparatus and Remote-SideStorage Apparatus>

When the asynchronous remote copy cannot be continued due to a networkfailure between the local-side storage apparatuses 1000L, 1000R and theremote-side storage apparatuses 2550L, 2550R, the following processingmay be performed.

(Processing of local-side storage apparatus) When the local-side storageapparatuses 1000L, 1000R detect a status where remote copy cannot becontinued, they record the write location recorded in the updateinformation of the journal data stored in the journal volumes 38320J1,38320J2 of the local-side storage apparatuses 1000L, 1000R in thedifferential bitmap (primary) of the local-side storage apparatuses1000L, 1000R. Further, when the local-side storage apparatuses 1000L,1000R receive a write request, they record the write location in thedifferential bitmap together with normal write processing.

(Processing of remote-side storage apparatus) When the remote-sidestorage apparatuses 2550L, 2550R detects a status where remote copycannot be continued, they release the journal data stored in the journalvolumes 38320J3, 38320J4 of the remote-side storage apparatuses 2550L,2550R (step 1). When the remote-side storage apparatuses 2550L, 2550Rreceive a write request, they record the write location in thedifferential bitmap (secondary) of the remote-side storage apparatuses2550L, 2550R together with normal write processing (step 2).Incidentally, foregoing step 1 and step 2 may be executed in independenttimings.

(Processing of local-side/remote-side storage apparatus upon receiving aresynchronization command) The remote-side storage apparatuses 2550L,2550R send data of the differential bitmap (secondary) to the local-sidestorage apparatuses 1000L, 1000R. The local-side storage apparatuses1000L, 1000R that received the differential bitmap confirm the contentsof the differential bitmap (primary) and the received differentialbitmap (secondary) so as to specify whether writing was made into thelocal-side storage apparatuses 1000L, 1000R or the remote-side storageapparatuses 2550L, 2550R after entering the Suspend status, andspecifies the write location when there is write data that has not yetbeen transferred to the remote-side storage apparatuses 2550L, 2550R.Subsequently, the local-side storage apparatuses 1000L, 1000R executedifferential copy regarding a specified location during theDuplex-Pending status.

Incidentally, the foregoing processing may also be executed based onsplit commands issued from the host 1100P. Incidentally, in theforegoing case, although there are cases of accessing the volumes of theremote-side storage apparatuses 2550L, 2550R after entering the Suspendstatus, the method of identifying the volumes to be used is the same asin the failover processing. However, in the failover processing, data ofan old volume is overwritten with data of a new volume, and allreplications will be lost in the remote copy of the remote-side storageapparatuses 2550L, 2550R in the subsequent differential copy. In orderto avoid this kind of situation, synchronous remote copy of theremote-side storage apparatuses 2550L, 2550R accompanying the failoverprocessing may be omitted.

<10. Variation of Asynchronous Remote Copy>

Other modes for realizing asynchronous remote copy in addition to themodes explained above may be considered.

<10.1. Grouping Mode of Write Data>

The local-side storage apparatuses 1000L, 1000R group and transfer oneor more write data. As the trigger for starting the collection of writedata into a new group, the lapse of a certain period of time from thestart of collection of the current group, and the local-side storageapparatuses 1000L, 1000R detecting that writing of a given volume hasbeen performed may be considered, but other triggers (for instance, uponreceiving a command from the host 1100P). Incidentally, unlike journaldata, the write data in the group do not have sequence information, butthe groups have sequence information. Further, when a plurality ofwritings update the same address, only the latest write data in the samegroup needs to be transferred.

By restoring only the write data in the group that satisfies both of thefollowing conditions, the remote-side storage apparatuses 2550L, 2550Rare able to protect the write sequence of data of the remote-sidestorage apparatus 2550 even without the sequence relationship in thegroup.

(A) All write data in the group are received by the remote-side storageapparatuses 2550L, 2550R.

(B) Further, write data of groups having an update number before thegroup of (A) are restored.

Incidentally, in order to perform the failover processing describedlater, the remote-side storage apparatuses 2550L, 2550R retain theupdate number of the restored groups according to a method, forinstance, described in the specification of US Patent Publication No.2005/0213389.

<10.2. Mode of Using Logical Snapshot>

The local-side storage apparatuses 1000L, 1000R repeatedly create alogical snapshot, and remotely write the data of such snapshot into thevolume of the remote-side storage apparatuses 2550L, 2550R. As explainedin the first to sixteenth embodiments, since the logical snapshotrecords the writing location pursuant to the Copy-On-Write processing,it is possible to identify the data to be remotely written.

Incidentally, in this remote writing, since the sequence relationship isnot retained in write request units as with Section 9.1, a logicalsnapshot (save snapshot) is also created for the volume of theremote-side storage apparatuses 2550L, 2550R, and remote writing isreceived thereafter. Further, the local-side storage apparatuses 1000L,1000R allocate an update number to the created logical snapshot toassociate with remote writing, and the remote-side storage apparatuses2550L, 2550R have information for determining for which generation(update number) the logical snapshot (save snapshot) was created. Thistechnology is explained in detail in the specification of US PatentPublication No. 2005/0210210.

<10.3. Failover Processing>

In both modes explained in Section 10.1 and Section 10.2, the sequencerelationship of the write data is unclear at the time the write data istransferred to the remote-side storage apparatuses 2550L, 2550R, and theidentification of the latest data accompanying the failover cannot beperformed. The following methods can be considered as countermeasures.

(A) The local-side both storage apparatuses 1000L, 1000R switch (createa snapshot) the groups of both storage apparatuses 1000L, 1000Ratomically through mutual coordination. Incidentally, atomic means thatthe write request is not processed during the switching (or creation) ofboth local-side storage apparatuses 1000L, 1000R, and this can berealized by the host 1100P or the local-side storage apparatus 1000L ofthe primary system reserving the write request. Further, the local-sideboth local-side storage apparatuses 1000L, 1000R allocate the sameupdate number to the switched group (or created snapshot). Incidentally,coordination of both local-side storage apparatuses 1000L, 1000R can berealized by the local-side storage apparatus 1000L of the local-sideprimary system issuing a command to the local-side storage apparatus1000R of the secondary system, but this coordination can also berealized by the resident software in the host 1100P periodically issuinga command to both local-side storage apparatuses 1000L, 1000R.

(B) In the failover processing, which remote-side storage apparatus2550L, 2550R has the latest data is determined by comparing the updatenumbers of groups (or logical snapshot (save snapshot)) in substitutefor the restored journal data update number.

<11. Variation of Failover Processing>

<11.1. When Clock is Assigned to Write Request from Host>

When the host 1100P is a mainframe, the time allocated by the host canbe associated with the write request as a rule of the I/O protocol, andthe sequence relationship of the write requests can be determined byusing this time. Therefore, by making the following changes oradditions, failover processing can be realized based on the writeassociated time.

(A) In the JNL creation processing, the write associated time isincluded in the update information at the time of creating journal data,and transferred to the remote-side storage apparatuses 2550L, 2550R.

(B) In the restoration processing, the latest write associated time ofthe restored journal data is stored.

(C) In the failover processing, the latest data is specified using thewrite associated time in substitute for the update number of journaldata.

<11.2. Comparison with Written Data of Application>

In normal synchronous remote copy, since it is not necessary toassociate the update time with the remote write request, this portionmust be expanded in the modes described above. In this Section, a modethat does not require such expansion is explained.

When the application 2010 (FIG. 38) operating in the host 1100P is adatabase, a log created by the database to be written into the volume isallocated with a transaction sequence number allocated by the database.Further, when re-booting the database, the database reads the log inorder to secure the atomicity of the transaction, and user of thedatabase will be able to know the latest transaction sequence numberthat has been committed up to that point in time.

Needless to say, since the transaction sequence number and log arewritten into the volume with a normal write request, they will become atarget of asynchronous remote copy. Therefore, if the failoverprocessing program 41100 (or administrator) performs the followingroutine, it will be possible to know which remote-side storage apparatus2550L, 2550R has the latest data.

(Step 1) The failover processing program 41100 issues a command to thedatabase of the host 1100B to re-boot using the volume of one of theremote-side storage apparatuses 2550L, 2550R. The database that receivedthe command performs the re-boot, and returns the latest transactionsequence number to the administrator or the failover processing program41100. Incidentally, the database may thereafter be ended once.

(Step 2) The failover processing program 41100 issues a command to thedatabase of the host 1100B to re-boot using the volume of the otherremote-side storage apparatus 2550L, 2550R. The database that receivedthe command performs the re-boot as with Step 1, and returns the latesttransaction sequence number to the administrator or the failoverprocessing program 41100.

(Step 3) The failover processing program 41100 compares the latesttransaction sequence numbers obtained at Step 1 and Step 2, anddetermines that the remote-side storage apparatus 2550L, 2550R used bythe database is the latest data when it is possible to return a newvalue.

Incidentally, this method is not limited to a database, if the I/O pathmanager 5000 (FIG. 5) or the file system 5020 (FIG. 5) is to create alog with an update sequence to a specific area of the volume, such logmay be used for the comparison. Incidentally, although the modeexplained in this Section does not require the expansion of thesynchronous remote copy, since it will depend on the type ofapplication, this mode does not deny the best modes explained above, noris this mode denied by the foregoing best modes.

<12. Case of Intersite Synchronous Remote Copy>

Incidentally, although data copy between the local-side storageapparatuses 1000L, 1000R and the remote-side storage apparatuses 2550L,2550R was performed with asynchronous remote copy in the foregoingexplanation, this may be substituted with synchronous remote copy.Incidentally, in the case of intersite synchronous remote copy, writecompletion is returned to the host 11000P at the time data copy orwriting to all four storage apparatuses (local-side storage apparatuses1000L, 1000R and remote-side storage apparatuses 2550L, 2550R) iscomplete. Nevertheless, since there are cases where one side of theintersite synchronous remote copy will become a Failure Suspend statusdepending on the pattern of network failure, similar failover processing(when required, allocation of update information with the primarylocal-side storage apparatus 1000L) will be required.

(18) Eighteenth Embodiment

In the seventeenth embodiment, the intersite network use efficiency wasinferior because data written by the host 1100P of the local site 38000Pwas transferred to the secondary site 38000B from both the local-sidestorage apparatus 1000L of the primary system and the local-side storageapparatus 1000R of the secondary system since processing of existingasynchronous remote copy is diverted considerably. In this embodiment,the method of improving the use efficiency by expanding asynchronousremote copy is described below.

<1. Overview of Present Embodiment>

FIG. 52 to FIG. 54 are schematic diagrams showing the invention of thisembodiment. Incidentally, the hardware constitution used in thisembodiment is similar to the seventeenth embodiment. Further, theprograms running in the hosts 1100P, 1100B and the respective storageapparatuses (local-side storage apparatuses 1000L, 1000R and remote-sidestorage apparatuses 2550L, 2550R) have the same constitution as theseventeenth embodiment. Nevertheless, since certain processing contentsof individual programs differ, the corresponding programs will bedescribed later.

<1.1. Normal Status>

FIG. 52 shows the normal status of the information system of thisembodiment. In this embodiment, as in the seventeenth embodiment, thewrite data sent from the I/O path manager 5000 (FIG. 5) in the host1100P to the primary local-side storage apparatus 1000L is changed intojournal data by the local-side and remote-side storage apparatuses1000L, 1000R. Nevertheless, although the primary local-side storageapparatus 1000L transfers the journal data to the remote-side primarystorage apparatus 2550L, journal is not normally transferred between thelocal-side secondary storage apparatus 1000R and the secondaryremote-side storage apparatus 2550R. Such being the case, data writteninto the remote-side primary storage apparatus 2550L is copied to thesecondary remote-side storage apparatus 2550R based on remote copy.

Incidentally, for the intersite transfer of journal data, the mode basedon a journal read request is considered first as in the seventeenthembodiment, but an asynchronous RC mode may also be adopted. In order todecide the transfer destination of journal data and to avoidunauthorized access and malfunctions, the respective local-side storageapparatuses 1000L, 1000R and the respective remote-side storageapparatuses 2550L, 2550R use information of the storage apparatusesregistered in the primary/secondary information 41340P, 41340B. Releaseof the journal data of the primary local-side storage apparatus 1000L isconducted based on the update number of the restored journal dataassociated with the journal read request. Further, release of thejournal data of the local-side secondary storage apparatus 1000R isconducted with the secondary journal release processing (not shown) as apart of the asynchronous remote copy program 41050P booted in thesecondary storage apparatus 1000R.

Further, remote copy (arrow 52010) of the data volumes 38310D5 to38310D8 between the secondary remote-side storage apparatuses 2550L,2550R may be synchronous remote copy or asynchronous remote copy (samenumber as the update number used in the intersite asynchronous remotecopy may be allocated to journal data of asynchronous remote copy of theremote-side storage apparatuses 2550L, 2550R). Further, a mode ofasynchronous remote copy known as differential remote copy may also beadopted.

Differential remote copy is a mode of recording the location of thewrite request to the copy source volume in the differential bitmap, andcopying the data of the location in the copy source volume to the copydestination volume upon finding the update the data transfer processingprogram (not shown), which is periodically operated, finding the updatelocation. Unlike other asynchronous remote copy, since this is similarto the format of synchronous remote copy where initialization copy andresynchronization copy are constantly operating, the processing issimple and the necessary amount of control information is small, but thewrite sequence of the copy destination volume cannot be protected whenthe storage apparatus having the copy source volume fails and stops.

<1.2. Status After Failure of Primary Site>

FIG. 54 is a schematic diagram showing the state after a failure occursat the local site in a normal status illustrated in FIG. 52. Theremote-side primary storage apparatus 2550L stops the restorationprocessing triggered by the processing request from the failoverprocessing program 41100 of the host 1100B, copies all data of theupdate location to be copied based on differential remote copy, ensuresthat the contents of the data volumes 38310D5 to 38310D8 of bothsecondary remote-side storage apparatuses 2550L, 2550R are uniform, andthereafter switches the copy mode to synchronous remote copy.Subsequently, the application processing is resumed while maintainingthe high availability constitution as in the other embodiments.Incidentally, write data from the host 1100B may be recorded with theupdate location based on a differential bitmap of intersite asynchronousremote copy. Further, when the local-side storage apparatuses 1000L,1000R of the local site 38000P become available (or the reason offailover is a plan failover and the local-side storage apparatuses1000L, 1000R enter their original operable state), journal data may becreated to operate the JNL creation processing in order to performasynchronous remote copy from the remote site 38000B to the local site38000P in the future.

<1.3. Status After Failure of Local-Side Primary Storage Apparatus>

FIG. 53 is a schematic diagram showing a state after the primarylocal-side storage apparatus 1000L fails and stops. The I/O path manager5000 (FIG. 5) of the host 1100P detects the failure and switches the I/Orequest destination to the secondary storage apparatus 1000R. Then,pursuant to this switch, the local-side secondary storage apparatus1000R becomes a new primary storage apparatus, and the primary/secondaryinformation 41340P (FIG. 41) is thereby updated.

Further, the storage apparatus 1000R that became the new local-sideprimary notifies the remote-side primary storage apparatus 2550L, whichis the transfer destination of journal data, that the primary system andthe secondary system has been changed, and further acquires the updatenumber of journal data in which the transfer is requested by theremote-side primary storage apparatus 2550L. Since the storage apparatus1000R that became a new local-side primary at the time of a normalstatus has created journal data, asynchronous remote copy can becontinued by transferring the journal data from the acquired updatenumber. Thus, even when the primary system and the secondary system areswitched pursuant to the failure of the primary local-side storageapparatus L, it is possible to preserve the write sequence of the datavolumes 38310D5, 38310D6 of the remote-side storage apparatus 2550L.

<2. Initialization of Information System>

Initialization of remote copy of the information system according tothis embodiment is performed in the following sequence. Incidentally,the argument of command and the path provided to the respectivelocal-side storage apparatuses 1000L, 1000R and the respectiveremote-side storage apparatuses 2550L, 2550R are the same as in theseventeenth embodiment.

(Step 1) Synchronous remote copy is performed from the data volumes38310D1, 38310D2 of the primary local-side storage apparatus 1000L tothe data volumes 38310D3, 38310D4 of the secondary storage apparatus1000R, and the routine waits until the initialization copy is complete.Setting of the synchronous remote copy is the same as in the seventeenthembodiment.

(Step 2) Asynchronous remote copy is performed from the data volumes38310D1, 38310D2 of the primary local-side storage apparatus 1000L tothe data volumes 38310D5, 38310D6 of the remote-side primary storageapparatus 2550L, and the routine waits until the initialization copy iscomplete. Incidentally, the local-side secondary storage apparatus 1000Rcreates journal data when the primary local-side storage apparatus 1000Lstarts creating journal data.

(Step 3) Remote copy is performed from the remote-side primary storageapparatus 2550L to the remote-side storage apparatus 2550R, and theroutine waits until the initialization copy is complete.

(Step 4) The apparatus number and journal group number of theprimary/secondary information 41340P, 41340B in the respectivelocal-side storage apparatuses 1000L, 1000R and the respectiveremote-side storage apparatuses 2550L, 2550R are updated.

The initialization routine is as described above. Incidentally, as inthe first to seventeenth embodiments, the setting of remote copy may bedesignated by the host 1100P, and the transition between steps and thesetting of remote copy may be performed with a program in the host11000P.

<3. Processing of I/O Path Manager>

In this embodiment, a trigger for switching the intersite asynchronousremote copy is necessary. This Section explains the required expansionin the foregoing case.

FIG. 55 and FIG. 56 are flowcharts explaining the expansion required inthe switch processing of the read request processing and the writerequest processing explained with reference to FIG. 10 and FIG. 11 ofthe first embodiment. Expansion is realized by inserting the followingprocessing between the determination of secondary system availabilityand inversion of remote copy. Incidentally, the processing at S55002,S55005 to S55008 of FIG. 55 is the same as the processing at S10011,S10012 to S10015 of FIG. 10, and the processing at S56002, S56005 toS56008 of FIG. 56 is the same as the processing at S1011, S11012 toS1015 of FIG. 11, and the explanation thereof is omitted.

(Update of primary/secondary information: Corresponds to S55003 of FIG.55 and S56003 of FIG. 56) The I/O path manager 5000 (FIG. 5) issues acommand to the local-side secondary storage apparatus 1000R to updatethe primary/secondary information 41340P (FIG. 41). The local-sidesecondary storage apparatus 1000R that received the command switches therelationship of the primary system and secondary system of both primarylocal-side storage apparatuses 1000L, 1000R shown with theprimary/secondary information 41340P, and updates information showingthat it is the primary storage apparatus.

Incidentally, it is also necessary to notify the local-side oldlocal-side storage apparatus 1000L and both secondary remote-sidestorage apparatuses 2550L, 2550R that the relationship of the primarysystem and secondary system of both primary local-side storageapparatuses 1000L, 1000R has been switched. As this method, the host1100P may directly communicate with the respective local-side storageapparatuses 1000L, 1000R and the respective remote-side storageapparatuses 2550L, 2550R, or the local-side secondary storage apparatus1000R may represent the other apparatuses and receive commands from thehost 1100P, and the secondary storage apparatus 1000R may distribute thecommands to the remaining storage apparatuses (local-side storageapparatus 1000R and the respective remote-side storage apparatuses2550L, 2550R). With this method, the local-side secondary storageapparatus 1000R that received the command from the host 1100P mayfurther distribute the command to the remote-side primary storageapparatus 2550L, and the remote-side primary storage apparatus 2550L mayadditionally distribute the command to the secondary remote-side storageapparatus 2550R. Further, as a method of issuing a command to thelocal-side old local-side storage apparatus 1000L, such command may besent together with the inversion command of synchronous remote copy.

(Switch of asynchronous remote copy: Corresponds to S55003 of FIG. 55and S56003 of FIG. 56) The I/O path manager 5000 (FIG. 5) issues aswitch command (this is hereinafter referred to as an “asynchronousremote copy switch command”) of the copy source of asynchronous remotecopy to the local-side secondary storage apparatus 1000R. When thelocal-side secondary storage apparatus 1000R receives the asynchronousremote copy switch command, it calls the switch processing program (notshown) of the asynchronous remote copy program 41050P, and it therebypersonally becomes the copy source of asynchronous remote copy.Incidentally, the identification number and journal group number of theremote-side primary storage apparatus 2550L and the journal group numberof the local-side secondary storage apparatus 1000R may be associatedwith the asynchronous remote copy switch command. In addition, theidentifying information of the data volumes 38310D1 to 38310D4 belongingto the local-side journal groups 38300G1, 38300G2 and the identifyinginformation of the data volumes 38310D5 to 38310D8 belonging to theremote-side journal groups 38300G3, 38300G4 configuring a pair may beassociated.

Incidentally, update of the primary/secondary information 41340P, 41340B(FIG. 41) and the asynchronous remote copy switch command may also beinserted before the synchronous remote copy inversion processing of FIG.39 (switch processing triggered by the local-side secondary storageapparatus 1000R receiving the write request).

<4. Asynchronous Remote Copy>

<4.1. Switch Processing>

The switch processing of the asynchronous remote copy programs 41050P,41050B is processing of switching the storage apparatuses 1000L, 1000Rto become the copy source triggered by a command from the host 11000P.Processing of the respective local-side storage apparatuses 1000L, 1000Rand remote-side storage apparatuses 2550L, 2550R involved in this switchprocessing is now explained. Incidentally, The adjectives of primary andsecondary used in the local-side storage apparatuses 1000L, 1000R andthe remote-side storage apparatuses 2550L, 2550R to execute thisprocessing may change due to the update of the primary/secondaryinformation 41340P, 41340B to be executed together with this processing.Therefore, the storage apparatuses (local-side storage apparatuses1000L, 1000R and remote-side storage apparatuses 2550L, 2550R) referredto as primary and secondary together with the term “old” in this Sectionshall be of a relationship before the update of the primary/secondaryinformation 41340P, 41340B based on the switch processing of the host1100P and before the execution of this processing.

<4.1.1. Processing of Local-Side Old Secondary Storage Apparatus>

FIG. 57 is a flowchart representing the processing contents of switchprocessing to be performed in the local-side old secondary storageapparatus 1000R that received a switch command from the host 1100P. Theprocessing contents are explained below with reference to the flowchart.

(S57001) The local-side old secondary storage apparatus 1000R receivesan asynchronous remote copy switch command sent from the host 11000P.

(S57002) The local-side old secondary storage apparatus 1000R confirmsthe argument of the received command.

(S57003) The local-side old secondary storage apparatus 1000R stops thesecondary journal release processing.

(S57004) The local-side old secondary storage apparatus 1000R sends thecommand received at S57001 to the remote-side primary storage apparatus2550L, and waits for the return of the update number of journal datathat needs to be transferred to the journal group 38300G3 of theopponent storage apparatus (in other words, the remote-side primarystorage apparatus 2550L) as the return value. Incidentally, when therequested journal group 38300G3 does not exist or the copy destinationdata volumes 38310D5, 38310D6 belonging to the journal volume 38300J3are of a Duplex-Pending status, since an error value will be obtainedinstead of the update number, an abnormal end is return to the host1100P in this processing, and this processing is thereby ended.

(S57005) The local-side old secondary storage apparatus 1000R updatesthe opponent storage apparatus number and the journal group number inthe journal group information 41330P to the apparatus number of theremote-side primary storage apparatus 2550L and journal group number ofthe journal group 38300G3 of the primary storage apparatus 2550L. As aresult of this update, it is possible to deny unauthorized journal readrequest from a storage apparatus other than the copy destination.

(S57006) The local-side old secondary storage apparatus 1000R registersand updates the asynchronous remote copy pair to be created in the copypair information 41300P.

(S45007) The local-side old secondary storage apparatus 1000R boots theJNLRD processing program 38120 and prepares to return a normal reply tothe journal read request.

<4.1.2. Remote-Side Primary Storage Apparatus>

FIG. 58 is a flowchart representing the processing contents of switchprocessing to be performed in the remote-side primary storage apparatus2550L. The processing contents are explained below with reference to theflowchart.

(S58001) The remote-side primary storage apparatus 2550L receives anasynchronous remote copy switch command sent by the local-side oldsecondary storage apparatus 1000R at S57004.

(S58002) The remote-side primary storage apparatus 2550L confirmswhether the switch command is from the local-side old secondary storageapparatus 1000R. Incidentally, when the source is other than thelocal-side old secondary storage apparatus R, this processing is ended.

(S58003) The remote-side primary storage apparatus 2550L confirms theexistence of the requested journal group 38300G3 and the data volumes38310D5, 38310D6. Incidentally, if the existence cannot be confirmed, anerror value is returned, and this processing is ended.

(S58004) The remote-side primary storage apparatus 2550L returns theupdate number of the restored journal data to the local-side oldsecondary storage apparatus 1000R.

(S58005) The remote-side primary storage apparatus 2550L registers andupdates the asynchronous remote copy pair to be created in the copy pairinformation 41300.

(S58006) The remote-side primary storage apparatus 2550L updates theopponent storage apparatus number and the journal group number in thejournal group information 41330P to the apparatus number of thelocal-side old secondary storage apparatus 1000R and the group number ofthe journal group 38300G3 of the old secondary storage apparatus 1000R.Thereby, the JNLRD processing program 38140 is able to switch thedestination of the journal read request to the local-side old secondarystorage apparatus 1000R.

<4.1.3. Local-Side Old Primary Storage Apparatus>

FIG. 59 is a flowchart representing the processing contents of switchprocessing to be performed in the local-side old local-side storageapparatus 1000L. The processing contents are explained below withreference to the flowchart.

(S59001) The local-side old local-side storage apparatus 1000L detectsthat the local-side old secondary storage apparatus 1000R became aprimary system. Incidentally, the following methods may be considered asthe detection method, but the method is not limited thereto.

(Method 1) An update command of the primary/secondary information 41340Pis received from the local-side old secondary storage apparatus 1000R.

(Method 2) A return value of the remote writing sent to the local-sideold secondary storage apparatus 1000R is detected.

(S59002) The local-side old local-side storage apparatus 1000L stops theJNLRD processing program 38120.

(S59003) The local-side old local-side storage apparatus 1000L boots thesecondary journal release processing.

<4.2. Journal Creation Processing>

In this embodiment also, as with the first to seventeenth embodiments,the primary local-side storage apparatus 1000L creates journal dataincluding the update number that it personally created, and thesecondary storage apparatus 1000R creates journal data based on theupdate information and update time contained in the remote writing.

<4.3. JNLRD Processing>

The processing explained with reference to FIG. 50 is also used in thisembodiment.

<4.4. JNLRD Processing>

In this embodiment, immediately after the start of processing accordingto the flowchart shown in FIG. 48, expansion is implemented so as toforemost confirm that the source of the journal read request is theremote-side primary storage apparatus 2550L registered in theprimary/secondary information 41340P or the journal group information41330P. There are the following reasons for adding this kind ofconfirmation processing.

(Reason 1) In order to prevent the leakage of data due to unauthorizedaccess; and

(Reason 2) If the journal read request is processed even when it is anunauthorized access, the update information transfer start address ofthe journal group information 41330P indicating the journal to besubsequently transferred will advance, and the journal data transferwill become inconsistent.

<4.5. Secondary Journal Release Processing>

FIG. 60 shows a flowchart representing the processing contents ofsecondary journal release processing. The processing contents areexplained below with reference to the flowchart.

(S60001) The secondary journal release processing acquires the restoredlatest update number of the journal group information 41330P related tothe primary local-side storage apparatus 1000L.

(S60002) The secondary journal release processing releases the journaldata up to the received update number. The specific method of release isthe same as the method explained in the JNLRD processing.

<5. Failover Processing>

Since the copy path to the secondary remote-side storage apparatus 2550Ris different in the seventeenth embodiment and this embodiment, failoveris conducted with processing that is different from FIG. 45. FIG. 61shows a flowchart representing the processing contents of failoverprocessing based on the failover processing program 41100 in thisembodiment.

(S61001) Foremost, the failover processing program 41100 (FIG. 53) ofthe secondary host 1100B issues a command to stop the JNLRD processingprogram 38140 of the remote-side storage apparatuses 2550L, 2550R viathe function I/F or the like. The remote-side storage apparatuses 2550L,2550R that received the command stop the JNLRD processing. Incidentally,his step may be omitted when the JNLRD processing program 38140 isautomatically stopped based on the asynchronous remote copy function.

(S61002) Subsequently, the failover processing program 41100 confirmsthat the remote-side primary storage apparatus 2550L completed therestoration processing based on the restoration processing program38130. For example, the secondary host 1100B issues a command to theremote-side storage apparatus 2550L via the function I/F to return acompletion notice at the time the restoration processing is complete.

(S61003) Subsequently, the failover processing program 41100 unifies thedata of the data volumes 38310D7, 38310D8 of the secondary remote-sidestorage apparatus 2550R with the data of the primary storage apparatus2550L. The unification method will differ as follows depending on theremote copy mode set between the remote-side storage apparatuses 2550L,2550R.

(Case of synchronous remote copy) Special processing is not requiredsince write data contained in the restored journal data is copied to theremote-side storage apparatus 2550R in the normal status.

(Case of asynchronous remote copy) The method waits for journal datathat has not been released to the journal volume 38320J3 of the primarystorage apparatus 2550L to exist.

(Case of differential remote copy) All differential bitmaps of theprimary storage apparatus 2550L are cleared.

(S61004) Subsequently, the failover processing program 41100 uses thefunction I/F or the like to form a synchronous remote copy pair with thedata volumes 38310D5, 38310D6 existing in the journal group 38300G3 ofthe selected remote-side storage apparatus 2550L as the primary system.Incidentally, this step can be omitted if synchronous remote copy hasalready been set from the normal status. Further, since the datacontents of the primary and remote-side storage apparatuses 2550L, 2550Rare the same even in the other remote copy modes, the initializationcopy and resynchronization copy for synchronous remote copy can beomitted.

(S61005) Subsequently, the failover processing program 41100 issues acommand to the application operating in the secondary host 1100B tostart access to the data volumes 38310D5, 38310D6 of the remote-sideprimary storage apparatus 2550L. Incidentally, the subsequent processingis the same as the other embodiments, including the first embodiment(first to seventeenth embodiments).

<6. Variation of Asynchronous Remote Copy>

As explained in the seventeenth embodiment, the asynchronous remote copymodes respectively have variations.

<6.1. Journal Data Transfer Mode Led by Local-Side Storage>

Although the secondary remote-side storage apparatuses 2550L, 2550R ledthe journal data transfer in the journal read request mode, in thismode, the primary local-side storage apparatuses 1000L, 1000R lead thejournal transfer. Here, if the following situations occursimultaneously, both local-side storage apparatuses 1000L, 1000R of thelocal-side primary system and secondary system will send journal data tothe remote-side primary storage apparatus 2550L, and inconsistency mayarise.

(Situation 1) The network between the primary local-side storageapparatus 1000L and the host 11000P and the secondary storage apparatus1000R is interrupted, but the network from the primary local-sidestorage apparatus 1000L to the remote-side primary storage apparatus2550L is communicable.

(Situation 2) The host 1100P sends a write request to the primarylocal-side storage apparatus 1000L in the state of Situation 1 and therequest destination is switched to the local-side secondary storageapparatus 1000R (in other words, the secondary system became the primarysystem).

(Situation 3) Although in Situation 2 a command should be issued to thelocal-side storage apparatus 1000L of the primary system to change to asecondary system, since the communication is interrupted, the local-sidestorage apparatus 1000L of the primary system is not changed to asecondary system, and consequently two apparatuses will become a primarysystem.

Thus, the remote-side primary storage apparatus 2550L confirms thesource upon receiving the journal data, and only receives journal datafrom the local-side storage apparatus 1000R designated as being alocal-side primary system with the asynchronous remote copy switchcommand or the like.

<6.2. Grouping Mode of Write Data>

As explained in the seventeenth embodiment, the grouped write data istreated as one journal data, and the processing disclosed in thisembodiment is performed.

<6.3. Snapshot Mode>

As explained in the seventeenth embodiment, the same generation numberis given to the snapshots created at the same timing in both primarylocal-side storage apparatuses 1000L, 1000R. Incidentally, in this mode,it is necessary to create a save snapshot in the secondary remote-sidestorage apparatuses 2550L, 2550R, and a snapshot may be created in theprimary storage apparatus 2550L as the creation destination. In the caseof this method, save snapshot data is written back (only differential)to the data volume of the copy destination based on the failoverprocessing program 41100 of the secondary host 11000B, and the datacontents of the primary storage apparatus 2550L and the remote-sidestorage apparatus 2550R are subsequently unified.

Incidentally, in addition to the above, a save snapshot may also becreated in the remote-side storage apparatus 2550R.

<7. Measures Against Failure in Primary Storage Apparatus of SecondarySite>

With the modes explained in various parts of this embodiment, there arecases when the secondary remote-side storage apparatus 2550R is not ableto take over asynchronous remote copy when the remote-side primarystorage apparatus 2550L fails and stops. This is because only theremote-side primary storage apparatus 2550L knows the data differentiallocation between the remote-side primary and remote-side storageapparatuses 2550L, 2550R. Therefore, in order for the secondaryremote-side storage apparatus 2550R to take over asynchronous remotecopy, the primary local-side storage apparatus 1000L needs to know up towhich journal data of the update number has reached the secondaryremote-side storage apparatus 2550R, and release journal data that areearlier than the arrived journal data. Two realization methods to matchthe remote copy mode between both secondary remote-side storageapparatuses 2550L, 2550R are explained below.

<7.1. Differential Remote Copy Mode>

<7.1.1. During Normal Operation>

The following processing is periodically repeated.

(Step 1) The remote-side primary storage apparatus 2550L stops therestoration processing.

(Step 2) The remote-side primary storage apparatus 2550L waits under thedata of the remote-side storage apparatus 2550R becomes uniform.

(Step 3) The remote-side primary storage apparatus 2550L stores theupdate number of journal data that was lastly restored during thestoppage at Step 1 as the value of the restored update number to bereturned to the primary local-side storage apparatus 1000L.

(Step 4) The remote-side primary storage apparatus 2550L resumes therestoration processing and waits for a given period of time.

Incidentally, the update number stored at Step 3 is conveyed to theprimary local-side storage apparatus 1000L based on the journal readrequest, and earlier journal data is released.

<7.1.2. During Failure in Remote Primary Storage Apparatus>

Upon detecting a failure in the remote-side primary storage apparatus2550L, the primary local-side storage apparatus 1000L records thejournal data into the write location by changing it into a differentialbitmap. Then, the primary local-side storage apparatus 1000L and the oldremote-side storage apparatus 2550R that newly became a primary systemon the remote side perform resynchronization processing of asynchronousremote copy with such differential bitmap.

<7.2. Asynchronous Remote Copy Mode>

<7.2.1. During Normal Operation>

The following processing is periodically repeated.

(Step 1) The remote-side primary storage apparatus 2550L restoresjournal data to the data volume 38300G3 of the copy destination.

(Step 2) The remote-side primary storage apparatus 2550L transfers thejournal data to the secondary remote-side storage apparatus 2550R basedon asynchronous remote copy.

(Step 3) The secondary remote-side storage apparatus 2550R restoresjournal data to the data volumes 38310D7, 38310D8 and sends the updatenumber of the restored journal data to the remote-side primary storageapparatus 2550L.

(Step 4) The remote-side primary storage apparatus 2550L releases thejournal data based on the update number received at Step 3, and furthersends the update number to the primary local-side storage apparatus1000L.

(Step 5) The primary local-side storage apparatus 1000L receives theupdate number of Step 4 and uses it to release the journal data.

<7.2.2. During Failure in Remote Primary Storage Apparatus>

After a failure occurs in the remote-side primary storage apparatus2550R, the primary local-side storage apparatus 1000L inquires thesecondary remote-side storage apparatus 2550R on the update number ofthe restored journal data to prepare for transferring from such journal.One secondary remote-side storage apparatus 2550R switches the source ofjournal data from the remote-side primary storage apparatus 2550L to theprimary local-side storage apparatus 1000L, and receives journal data.

<8. Variation>

In the modes explained above, the source of journal data was always thelocal-side storage apparatus 1000L, but the secondary storage apparatus1000R may also be used as the source.

1. An information system having a first computer, a first storageapparatus coupled to said first computer and including a first volume,and a second storage apparatus coupled to said first storage apparatusand said first computer and including a second volume, comprising: athird storage apparatus coupled to said first storage apparatus andincluding a third volume; and a fourth storage apparatus coupled to saidsecond storage apparatus and including a fourth volume; wherein saidfirst and second storage apparatuses execute remote copy of copying datastored in said first volume to said second volume; wherein said firstand third storage apparatuses execute remote copy of copying data storedin said first volume to said third volume; and wherein said second andfourth storage apparatuses execute remote copy of coping data stored insaid second volume to said fourth volume.
 2. The information systemaccording to claim 1, wherein said first storage apparatus assigns aunique update number to the data written from said first computer intosaid first volume, and notifies said corresponding update number to saidsecond or third storage apparatus upon copying said data to said secondor third volume; and wherein said second storage apparatus notifies saidcorresponding update number to said fourth storage apparatus uponcopying data of said second volume to said fourth volume.
 3. Theinformation system according to claim 2, wherein said third storageapparatus is coupled to said fourth storage apparatus; and wherein saidthird and fourth storage apparatuses start remote copy between saidthird and fourth volume with said third or fourth storage apparatusretaining latest data as the primary system when a failure occurs insaid first and/or second storage apparatus.
 4. The information systemaccording to claim 3, further comprising a second computer coupled tosaid third and fourth storage apparatus; wherein said second computercompares said update number of said data stored in said third volume,and said update number of said corresponding data stored in said fourthvolume; selects said third or fourth storage apparatus having latestdata based on the comparative result; and forms a copy pair with thirdand fourth volumes with said selected third or fourth storage apparatusas the primary system.
 5. The information system according to claim 1,wherein said first computer switches a path to said second storageapparatus when a failure occurs in said first storage apparatus.
 6. Theinformation system according to claim 1, wherein said first and secondstorage apparatuses perform remote copy of copying data stored in saidfirst volume to said second volume synchronously with the writing ofsaid data into said first volume; wherein said first and third storageapparatuses perform remote copy of copying data stored in said firstvolume to said third volume asynchronously with the writing of said datainto said first volume; and wherein said second and fourth storageapparatuses copy data stored in said second volume to said fourth volumeasynchronously with the writing of said data into second volume.
 7. Theinformation system according to claim 1, wherein said first storageapparatus sends said data to said third storage apparatus according to arequest from said third storage apparatus; and wherein said secondstorage apparatus sends said data to said fourth storage apparatusaccording to a request from said fourth storage apparatus.
 8. Theinformation system according to claim 1, wherein, upon receiving saiddata to be written into said first volume from said host, said firststorage apparatus creates journal data containing said data and updateinformation of said data including said update number, stores thecreated journal data in a first journal volume provided in said firststorage apparatus, and sends said data to said second storage apparatustogether with said update number; wherein, upon receiving said data fromsaid first storage apparatus, said second storage apparatus stores saiddata in said second volume, creates journal data containing said dataand update information of said data including said update numbernotified from said first storage apparatus, and stores the createdjournal data in a second journal volume provided in said second storageapparatus; wherein said third storage apparatus stores said journal datasent from said first storage apparatus in a third journal volumeprovided in said third storage apparatus, and stores said data containedin said journal data in said third volume; and wherein said fourthstorage apparatus stores said journal data sent from said second storageapparatus in a fourth journal volume provided in said fourth storageapparatus, and stores said data contained in said journal data in saidfourth volume.
 9. A data protection method in an information systemhaving a first computer, a first storage apparatus coupled to said firstcomputer and including a first volume, and a second storage apparatuscoupled to said first storage apparatus and said first computer andincluding a second volume, comprising: a first step of connecting athird storage apparatus including a third volume to said first storageapparatus and connecting a fourth storage apparatus including a fourthvolume to said second storage apparatus, and said first and secondstorage apparatuses executing remote copy of copying data stored in saidfirst volume to said second volume; and a second step of said first andthird storage apparatuses executing remote copy of copying data storedin said first volume to said third volume, and said second and fourthstorage apparatuses executing remote copy of copying data stored in saidsecond volume to said fourth volume.
 10. An information system having afirst computer, a first storage apparatus coupled to said first computerand including a first volume, and a second storage apparatus coupled tosaid first storage apparatus and said first computer and including asecond volume, comprising: a third storage apparatus coupled to saidfirst and second storage apparatuses and including a third volume; and afourth storage apparatus coupled to said third storage apparatus andincluding a fourth volume; wherein said first and second storageapparatuses execute remote copy of copying data stored in said firstvolume to said second volume; wherein said first and third storageapparatuses execute remote copy of copying data stored in said firstvolume to said third volume; and wherein said third and fourth storageapparatuses execute remote copy of coping data stored in said thirdvolume to said fourth volume.
 11. The information system according toclaim 10, wherein said first computer switches a path to said secondstorage apparatus when a failure occurs in said first storage apparatus.wherein said second storage apparatus writes data provided from saidcomputer into said second volume; and wherein said second and thirdstorage apparatuses copy said data stored in said second volume to saidthird volume.
 12. The information system according to claim 11, whereinsaid third and fourth storage apparatuses continue remote copy betweensaid third and fourth volumes even after a failure occurs in said firststorage apparatus.
 13. The information system according to claim 10,further comprising a second computer coupled to said third and fourthstorage apparatuses; wherein said second computer takes over theprocessing of said first computer and writes data into said third volumewhen a failure occurs in said first and/or second storage apparatus; andwherein said third and fourth storage apparatuses execute remote copybetween said third and fourth volumes with said third storage apparatusas the primary system.
 14. A data protection method in an informationsystem having a first computer, a first storage apparatus coupled tosaid first computer and including a first volume, and a second storageapparatus coupled to said first storage apparatus and said firstcomputer and including a second volume, comprising: a first step ofconnecting a third storage apparatus including a third volume to saidfirst and second storage apparatuses and connecting a fourth storageapparatus including a fourth volume to said third storage apparatus,said first and second storage apparatuses executing remote copy ofcopying data stored in said first volume to said second volume, and saidfirst and third storage apparatuses executing remote copy of copyingdata stored in said first volume to said third volume; and a second stepof said third and fourth storage apparatuses executing remote copy ofcopying data stored in said third volume to said fourth volume.